Proxmox Update 8 to 9 does not boot anymore - black screen

Problem

After I updated my Asus/Intel NUC from proxmox v8.4.14 to 9.1.6, it did not boot anymore. I followed the instructions of https://pve.proxmox.com/wiki/Upgrade_from_8_to_9 and used the update checker tool:

1. Update checker with pve8to9 
   --> all green ✅
2. Updated from 8.4.14 to 9.1.6 complete using this 
   --> GUI & CLI showed new version ✅
   --> VMs and Containers continued to run normally ✅
3. Reboot 
   --> does not boot anymore ❌
4. BIOS Update of the Asus/Intel NUC did not help ❌

Solution

I found Dustin Rues awesome blog entry: https://dustinrue.com/2025/12/recovering-from-a-failed-proxmox-upgrade/

1. Boot from live linux via USB stick (in my case linux mint)

2. In the live linux terminal, run the following commands:

mount /dev/mapper/pve-root /mnt
mount /dev/sda2 /mnt/boot/efi
mount --bind /dev /mnt/dev
mount --bind /sys /mnt/sys
mount --bind /proc /mnt/proc
mount --bind /dev/pts /mnt/dev/pts
chroot /mnt
mount -t efivarfs efivarfs /sys/firmware/efi/efivars
grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox /dev/sda
grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox --removable /dev/sda
echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' | debconf-set-selections -v -u
apt install --reinstall grub-efi-amd64
update-initramfs -u -k all

Once this finished I did the following

umount /mnt/dev/pts
umount /mnt/proc
umount /mnt/dev
umount /mnt/sys/firmware/efi/efivars
umount /mnt/sys
umount /mnt/boot/efi
umount /mnt/
reboot

Example

root@mint:~#
root@mint:~#
root@mint:~# mount /dev/mapper/pve-root /mnt
root@mint:~# mount /dev/sda2 /mnt/boot/efi/
root@mint:~# mount --bind /dev /mnt/dev
root@mint:~# mount --bind /sys /mnt/sys
root@mint:~# mount --bind /proc /mnt/proc
root@mint:~# mount --bind /dev/pts /mnt/dev/pts
root@mint:~#
root@mint:~# chroot /mnt
root@mint:/#
root@mint:/# mount -t efivarfs efivarfs /sys/firmware/efi/efivars/
root@mint:/#
root@mint:/# grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox /dev/sda
Installing for x86_64-efi platform.
Installation finished. No error reported.
root@mint:/#
root@mint:/# grub-install --target x86_64-efi --no-floppy --bootloader-id proxmox --removable /dev/sda
Installing for x86_64-efi platform.
Installation finished. No error reported.
root@mint:/#
root@mint:/#
root@mint:/# echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' | debconf-set-selections -v -u
info: Trying to set 'grub2/force_efi_extra_removable' [boolean] to 'true'
info: Loading answer for 'grub2/force_efi_extra_removable'
root@mint:/#
root@mint:/# apt install --reinstall grub-efi-amd64
The following packages were automatically installed and are no longer required:
  bsdmainutils                    libdav1d6               libllvm15               libpython3.9-stdlib  perl-modules-5.32                   python3-soupsieve
  gcc-12-base                     libdns-export1110       libmpdec3               librav1e0            perl-modules-5.36                   python3-talloc
  libabsl20220623                 libdrm-nouveau2         libnsl-dev              libsubid4            proxmox-kernel-6.8.12-1-pve-signed  python3-tempita
  libapt-pkg6.0                   libdrm-radeon1          libnumber-compare-perl  libsvtav1enc1        pve-kernel-5.15                     python3-tz
  libavif15                       libfile-find-rule-perl  libopts25               libtext-glob-perl    pve-kernel-5.15.126-1-pve           python3-waitress
  libboost-context1.74.0          libflac12               libperl5.32             libthrift-0.17.0     pve-kernel-5.15.158-2-pve           python3-webtest
  libboost-coroutine1.74.0        libfmt9                 libperl5.36             libtiff5             python3-bs4                         python3.11
  libboost-filesystem1.74.0       libglusterd0            libprocps8              libtirpc-dev         python3-jaraco.classes              python3.11-minimal
  libboost-iostreams1.74.0        libicu67                libprotobuf23           liburing1            python3-ldb                         python3.9
  libboost-program-options1.74.0  libicu72                libpython3.11           libwebp6             python3-paste                       python3.9-minimal
  libboost-thread1.74.0           libisc-export1105       libpython3.11-minimal   libx265-199          python3-pastedeploy                 sgml-base
  libbpf0                         libjs-sencha-touch      libpython3.11-stdlib    libxcb-dri2-0        python3-pastedeploy-tpl             telnet
  libcbor0                        libldap-2.5-0           libpython3.9            libzpool5linux       python3-pytz                        usrmerge
  libcbor0.8                      libleveldb1d            libpython3.9-minimal    lua-lpeg             python3-singledispatch
Use 'apt autoremove' to remove them.

Installing:
  grub-efi-amd64

REMOVING:
  grub-pc

Summary:
  Upgrading: 0, Installing: 1, Removing: 1, Not Upgrading: 0
  Download size: 46.7 kB
  Freed space: 349 kB

Continue? [Y/n]
Get:1 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 grub-efi-amd64 amd64 2.12-9+pmx2 [46.7 kB]
Fetched 46.7 kB in 0s (398 kB/s)          
Preconfiguring packages ...
(Reading database ... 119801 files and directories currently installed.)
Removing grub-pc (2.12-9+pmx2) ...
Selecting previously unselected package grub-efi-amd64.
(Reading database ... 119792 files and directories currently installed.)
Preparing to unpack .../grub-efi-amd64_2.12-9+pmx2_amd64.deb ...
Unpacking grub-efi-amd64 (2.12-9+pmx2) ...
Setting up grub-efi-amd64 (2.12-9+pmx2) ...
Installing for x86_64-efi platform.
File descriptor 3 (pipe:[43286]) leaked on vgs invocation. Parent PID 3806: grub-install.real
File descriptor 3 (pipe:[43286]) leaked on vgs invocation. Parent PID 3806: grub-install.real
Installation finished. No error reported.
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.17.13-1-pve
Found initrd image: /boot/initrd.img-6.17.13-1-pve
Found linux image: /boot/vmlinuz-6.8.12-17-pve
Found initrd image: /boot/initrd.img-6.8.12-17-pve
Found linux image: /boot/vmlinuz-6.8.12-1-pve
Found initrd image: /boot/initrd.img-6.8.12-1-pve
Found linux image: /boot/vmlinuz-5.15.158-2-pve
Found initrd image: /boot/initrd.img-5.15.158-2-pve
Found linux image: /boot/vmlinuz-5.15.126-1-pve
Found initrd image: /boot/initrd.img-5.15.126-1-pve
Found linux image: /boot/vmlinuz-5.4.203-1-pve
Found initrd image: /boot/initrd.img-5.4.203-1-pve
Found linux image: /boot/vmlinuz-5.4.73-1-pve
Found initrd image: /boot/initrd.img-5.4.73-1-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done
Processing triggers for man-db (2.13.1-1) ...
root@mint:/#
root@mint:/#
root@mint:/# 
root@mint:/# 
root@mint:/# 
root@mint:/# 
root@mint:/# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-6.17.13-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-6.8.12-17-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-6.8.12-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.158-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.15.126-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.4.203-1-pve
W: zstd compression (CONFIG_RD_ZSTD) not supported by kernel, using gzip
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.4.73-1-pve
W: zstd compression (CONFIG_RD_ZSTD) not supported by kernel, using gzip
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
root@mint:/#
root@mint:/#
root@mint:/#
root@mint:/# exit
exit
root@mint:~#
root@mint:~# ls /mnt
bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@mint:~#
root@mint:~# umount /mnt/dev/pts
root@mint:~# umount /mnt/proc
root@mint:~# umount /mnt/dev
root@mint:~# umount /mnt/sys/firmware/efi/efivars
root@mint:~# umount /mnt/sys
root@mint:~# umount /mnt/boot/efi
root@mint:~# umount /mnt/
root@mint:~#
root@mint:~# reboot

Azure Linux Ubuntu not fully upgraded

Using apt-get update && apt-get upgrade -y on your Ubuntu VM in Azure sometimes does not upgrade all packages:

root@hostname6:~#
root@hostname6:~#
root@hostname6:~# apt-get update && apt-get upgrade -y
Hit:1 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu jammy InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:4 http://azure.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:5 http://azure.archive.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
libnss-systemd libpam-systemd libsystemd0 libudev1 linux-azure linux-cloud-tools-azure linux-headers-azure linux-image-azure linux-tools-azure
systemd systemd-sysv udev
0 upgraded, 0 newly installed, 0 to remove and 12 not upgraded.
root@hostname6:~#

Solution

sudo apt-get install aptitude -y
sudo aptitude safe-upgrade

aptitude safe-upgrade
Upgrades installed packages to their most recent version. Installed packages will not be removed unless they are unused [...] Packages which are not currently installed may be installed to resolve dependencies unless the --no-new-installs command-line option is supplied.

Example

root@hostname6:~#
root@hostname6:~# sudo aptitude safe-upgrade
Resolving dependencies...
The following NEW packages will be installed:
linux-azure-6.8-cloud-tools-6.8.0-1041{a} linux-azure-6.8-headers-6.8.0-1041{a} linux-azure-6.8-tools-6.8.0-1041{a}
linux-cloud-tools-6.8.0-1041-azure{a} linux-headers-6.8.0-1041-azure{a} linux-image-6.8.0-1041-azure{a} linux-modules-6.8.0-1041-azure{a}
linux-tools-6.8.0-1041-azure{a}
The following packages will be upgraded:
libnss-systemd libpam-systemd libsystemd0 libudev1 linux-azure linux-cloud-tools-azure linux-headers-azure linux-image-azure linux-tools-azure
systemd systemd-sysv udev
12 packages upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
Need to get 67.2 MB of archives. After unpacking 269 MB will be used.
Do you want to continue? [Y/n/?]

[..]
Current status: 0 (-12) upgradable.
root@hostname6:~#
root@hostname6:~# apt-get update
Hit:1 http://azure.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://azure.archive.ubuntu.com/ubuntu jammy-security InRelease
Hit:5 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease
Reading package lists... Done
root@hostname6:~#
root@hostname6:~# apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@hostname6:~#
root@hostname6:~#

Disabled Transparent Huge Pages with created systemd service

In some applications (e.g. Splunk) you need to disable Transparent Huge Pages THP. 

When your systems are running in a managed environment (e.g. the public cloud - like Microsoft Azure) with cloud images (like "0001-com-ubuntu-server-jammy" or "0001-com-ubuntu-confidential-vm-jammy" (both Ubuntu22.04 LTS)), you may not be able to use the GRUB Edit to disable THP, because the cloud image ignores the GRUB edits. A possible solution can be a custom systemd service:

Example Disable Transparent Huge Pages once (not reboot-persistent💢):

user@devazubu227:~$
user@devazubu227:~$
user@devazubu227:~$ sudo cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
user@devazubu227:~$
user@devazubu227:~$
user@devazubu227:~$ echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
never
user@devazubu227:~$
user@devazubu227:~$ sudo cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
user@devazubu227:~$

Example Disable Transparent Huge Pages with custom systemd service (reboot-persistent ✅):

Commands:

sudo tee /etc/systemd/system/disable-thp.service > /dev/null <<EOF
[Unit]
Description=Disable Transparent Huge Pages
After=network.target

[Service]
Type=simple
ExecStart=/bin/bash -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl enable disable-thp
sudo systemctl start disable-thp
sudo systemctl status disable-thp
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

Example

user@devazubu227:~$
user@devazubu227:~$
user@devazubu227:~$ sudo tee /etc/systemd/system/disable-thp.service > /dev/null <<EOF
[Unit]
Description=Disable Transparent Huge Pages
After=network.target

[Service]
Type=simple
ExecStart=/bin/bash -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target
EOF
user@devazubu227:~$
user@devazubu227:~$ cat /etc/systemd/system/disable-thp.service
[Unit]
Description=Disable Transparent Huge Pages
After=network.target

[Service]
Type=simple
ExecStart=/bin/bash -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target
user@devazubu227:~$
user@devazubu227:~$
user@devazubu227:~$ sudo systemctl daemon-reexec
user@devazubu227:~$ sudo systemctl daemon-reload
user@devazubu227:~$ sudo systemctl enable disable-thp
Created symlink /etc/systemd/system/multi-user.target.wants/disable-thp.service → /etc/systemd/system/disable-thp.service.
user@devazubu227:~$ sudo systemctl start disable-thp

user@devazubu227:~$ sudo systemctl status disable-thp
○ disable-thp.service - Disable Transparent Huge Pages
Loaded: loaded (/etc/systemd/system/disable-thp.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Thu 2025-08-07 11:01:42 UTC; 11s ago
Process: 4394 ExecStart=/bin/bash -c echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defra>
Main PID: 4394 (code=exited, status=0/SUCCESS)
CPU: 2ms

Aug 07 11:01:42 devazubu227 systemd[1]: Started Disable Transparent Huge Pages.
Aug 07 11:01:42 devazubu227 systemd[1]: disable-thp.service: Deactivated successfully.
user@devazubu227:~$
user@devazubu227:~$
user@devazubu227:~$ cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
user@devazubu227:~$
user@devazubu227:~$ cat /sys/kernel/mm/transparent_hugepage/defrag
always defer defer+madvise madvise [never]
user@devazubu227:~$
user@devazubu227:~$


Surface 2 Pro Install Linux Ubuntu 24.04 LTS

Windows 10 support ended. Microsoft Surface 2 Pro devices can't officially be updated to Windows 11. This guide shows how to switch to Linux Ubuntu 24.04 LTS instead:

Installation steps 

1. Download Rufus https://github.com/pbatard/rufus
2. Download Linux, e.g. Ubuntu 24 https://ubuntu.com/download/desktop
3. Format USB stick with Rufus
   - Select the USB stick (all files on it will be erased)
   - Select image file
   - Select MBR
   - Select BIOS or UEFI
   - Select Fix for older BIOSe
   - Select FAT32
   - Press Start
   
   
   
4. Shutdown Surface 2 device
5. Hold Volume-Up & hold Home-Button until Surface logo appears
6. Disable Secure Boot --> Red background around Surface logo on boot
7. Shutdown Surface 2 device
8. Insert USB Stick into Surface 2
9. Hold Volume-Down & press Home-Button until Surface logo appears 
10. Select Install Ubuntu in GRUB loader 
11. Optional: For additional security (using encryption@rest) use lvm with encryption and use a long unique passphrase. 

Result 

Other Microsoft Surface devices might use: https://github.com/linux-surface/linux-surface/wiki/Installation-and-Setup
 

Powershell Website "Ping" using the winsystems web proxy

If you want to monitor/check (every 10s) continuesly if a website (in this exampe http://www.google.de) is reachable using powershell and also using the configured webproxy in your windows system, then you can use the following example:

while ($true) {
    $webClient = New-Object System.Net.WebClient
    $proxy = [System.Net.WebRequest]::GetSystemWebProxy()
    $proxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
    $webClient.Proxy = $proxy

    $proxyUri = $proxy.GetProxy([System.Uri]::new("http://www.google.de"))
    $proxyIp = $proxyUri.Host
    $proxyPort = $proxyUri.Port

    $startTime = Get-Date
    try {
        $response = $webClient.DownloadString("http://www.google.de")
        $endTime = Get-Date
        $latency = ($endTime - $startTime).TotalMilliseconds
        $statusCode = 200 # WebClient doesn't expose status code directly, assuming success here
        Write-Output "[$startTime] Latency: ${latency}ms, HTTP Status Code: $statusCode, Proxy:         $proxyIp :$proxyPort"
    } catch {
        $endTime = Get-Date
        $latency = ($endTime - $startTime).TotalMilliseconds
        $errorMessage = $_.Exception.Message
        Write-Output "[$startTime] Latency: ${latency}ms, Error: $errorMessage, Proxy: $proxyIp :$proxyPort"
    }

Start-Sleep -Seconds 10
}



Ansible Remote Shell Examples

To execute remote commands or get access to a remote server using ansible, you can do:

source =prdeu4spl002 
destination = prdus1ans105 
auth = ssh-keys already installed (not part of the example)
command1 = pwd
command2 = ls -lah
command3 = /usr/local/sbin/schedule-downtime -s "now" -d 1 -r "Test Downtime"

Example




[spluser@prdeu4spl002 ansible]$
[spluser@prdeu4spl002 ansible]$
[spluser@prdeu4spl002 ansible]$ ansible all -i ,prdus1ans105 -b -m shell -a 'pwd'
prdus1ans105 | CHANGED | rc=0 >>
/home/spluser
[spluser@prdeu4spl002 ansible]$

[spluser@prdeu4spl002 ansible]$ ansible all -i ,prdus1ans105 -b -m shell -a 'ls -lah'
prdus1ans105 | CHANGED | rc=0 >>
total 16K
drwx------. 5 spluser spluser 130 Jul 25 2024 .
drwxr-xr-x. 12 root root 183 May 8 15:20 ..
drwx------ 3 spluser spluser 17 Jun 25 2024 .ansible
-rw------- 1 spluser spluser 182 Jun 4 12:55 .bash_history
-rw-r--r--. 1 spluser spluser 18 Jun 20 2022 .bash_logout
-rw-r--r--. 1 spluser spluser 141 Jun 20 2022 .bash_profile
-rw-r--r--. 1 spluser spluser 376 Jun 20 2022 .bashrc
drwxr-x--- 5 spluser spluser 39 Jun 25 2024 .puppetlabs
drwx------. 2 spluser spluser 29 Jun 25 2024 .ssh
[spluser@prdeu4spl002 ansible]$
[spluser@prdeu4spl002 ansible]$
[spluser@prdeu4spl002 ansible]$ ansible all -i ,prdus1ans105 -b -m shell -a '/usr/local/sbin/schedule-downtime -s "now" -d 1 -r "Test Downtime"'
prdus1ans105 | CHANGED | rc=0 >>

Now scheduling a downtime from 14:59:02 until 15:00:02 for prdus1ans105 in Mon01 and nIcinga because of "Test Downtime" on behalf of user ROOT.

contacting the Mon01 satellite REST API...
OK

contacting the nIcinga REST API...
OK
[spluser@prdeu4spl002 ansible]$
[spluser@prdeu4spl002 ansible]$





Add a CA certificate to GitLab running in a podman container

Adding a CA certificate to GitLab which is running in a podman container (also works with docker containers, just replace podman with docker):

1. Login to the podman container 
2. Copy/install the CA certificates (in this case Digi-Issuing-CA01-G3.pem & Digi-Root-CA01-G3.pem)
3. Restart the podman container

Example

euprdgitlab655:~ #
euprdgitlab655:~ # podman exec -it gitlab /bin/bash
root@ad24f5df0102:/#
root@ad24f5df0102:/#
root@ad24f5df0102:/# ls /etc/gitlab/
gitlab-secrets.json gitlab.rb ssh_host_ecdsa_key ssh_host_ecdsa_key.pub ssh_host_ed25519_key ssh_host_ed25519_key.pub ssh_host_rsa_key ssh_host_rsa_key.pub ssl trusted-certs
root@ad24f5df0102:/#
root@ad24f5df0102:/# ls /etc/gitlab/trusted-certs/
07ac5923.0 Digi-Issuing-CA01-G2.pem Digi-Root-CA-G2.pem e0c0effb.0
root@ad24f5df0102:/#
root@ad24f5df0102:/# ls -lah /etc/gitlab/trusted-certs/
total 8.0K
drwxr-xr-x 2 root root 101 Mar 27 12:44 .
drwxrwxr-x 4 root root 250 Oct 15 2024 ..
lrwxrwxrwx 1 root root 19 Mar 27 12:44 07ac5923.0 -> Digi-Root-CA-G2.pem
-rw-r--r-- 1 root root 2.6K Sep 14 2021 Digi-Issuing-CA01-G2.pem
-rw-r--r-- 1 root root 2.3K Sep 14 2021 Digi-Root-CA-G2.pem
lrwxrwxrwx 1 root root 24 Mar 27 12:44 e0c0effb.0 -> Digi-Issuing-CA01-G2.pem
root@ad24f5df0102:/#
root@ad24f5df0102:/#
root@ad24f5df0102:/# vi /etc/gitlab/trusted-certs/Digi-Root-CA-G3.pem
root@ad24f5df0102:/# vi /etc/gitlab/trusted-certs/Digi-Issuing-CA-G3.pem
root@ad24f5df0102:/#
root@ad24f5df0102:/# ls -lah /etc/gitlab/trusted-certs/
total 16K
drwxr-xr-x 2 root root 158 Jul 14 10:16 .
drwxrwxr-x 4 root root 250 Oct 15 2024 ..
lrwxrwxrwx 1 root root 19 Mar 27 12:44 07ac5923.0 -> Digi-Root-CA-G2.pem
-rw-r--r-- 1 root root 2.3K Jul 14 10:16 Digi-Issuing-CA-G3.pem
-rw-r--r-- 1 root root 2.6K Sep 14 2021 Digi-Issuing-CA01-G2.pem
-rw-r--r-- 1 root root 2.3K Sep 14 2021 Digi-Root-CA-G2.pem
-rw-r--r-- 1 root root 1.9K Jul 14 10:16 Digi-Root-CA-G3.pem
lrwxrwxrwx 1 root root 24 Mar 27 12:44 e0c0effb.0 -> Digi-Issuing-CA01-G2.pem
root@ad24f5df0102:/#
root@ad24f5df0102:/# exit
exit
euprdgitlab655:~ #
euprdgitlab655:~ #
euprdgitlab655:~ #
euprdgitlab655:~ # podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ad24f5df0102 reg.subdomain.domain.tld/gitlab/gitlab-ee:18.0.3-ee.0 /assets/wrapper 3 months ago Up 3 weeks (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:2222->22/tcp gitlab
aa22bdf8c33a docker.io/library/nginx:1.27.5 nginx -g daemon o... 3 months ago Up 3 weeks 0.0.0.0:8443->8443/tcp nginx
euprdgitlab655:~ #
euprdgitlab655:~ # podman stop gitlab
WARN[0010] StopSignal SIGTERM failed to stop container gitlab in 10 seconds, resorting to SIGKILL
gitlab
euprdgitlab655:~ # podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ad24f5df0102 reg.subdomain.domain.tld/gitlab/gitlab-ee:18.0.3-ee.0 /assets/wrapper 3 months ago Up 2 seconds (starting) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:2222->22/tcp gitlab
aa22bdf8c33a docker.io/library/nginx:1.27.5 nginx -g daemon o... 3 months ago Up 3 weeks 0.0.0.0:8443->8443/tcp nginx
euprdgitlab655:~ #
euprdgitlab655:~ #

[...] *wait* [...]

euprdgitlab655:~ #
euprdgitlab655:~ # podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ad24f5df0102 reg.subdomain.domain.tld/gitlab/gitlab-ee:18.0.3-ee.0 /assets/wrapper 3 months ago Up 7 minutes (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:2222->22/tcp gitlab
aa22bdf8c33a docker.io/library/nginx:1.27.5 nginx -g daemon o... 3 months ago Up 3 weeks 0.0.0.0:8443->8443/tcp nginx
euprdgitlab655:~ #

Backup GitLab running in a Container and encrypt the backup

Many Gitlab instances run in a docker or podman container. The following is a bash script which 

1. fetches the gitlab-configs and the gitlab-data from inside of the container, 
2. then creates a key (symmetric), 
3. which is again encrypted with a asymmetric key (certificate)
4. and encrypted the gitlab-config & data with that
5. And it deletes old backup data which is older then 7 days

 

gitlab040:/srv/gitlab # cat gitlab-backup.sh
#!/bin/bash
# remove tmp files 
rm -rf /srv/gitlab/tmp

# create tmp directory
mkdir -p /srv/gitlab/tmp

# generate backups
docker exec -t gitlab /bin/sh -c 'umask 0077; tar cfz /secret/gitlab/backups/$(date +"%Y-%m-%d-%H-%M")_config_gitlab_backup.tgz -C / etc/gitlab'
docker exec -t gitlab gitlab-backup create CRON=1 BACKUP=$(date +"%Y-%m-%d-%H-%M")_data

# locate backup files
BACKUPCONFIG=$(ls -Art /srv/gitlab/backup-config/*config_gitlab_backup.tgz | tail -n 1)
BACKUPDATA=$(ls -Art /srv/gitlab/backup-data/*data_gitlab_backup.tar | tail -n 1)

# generate symmetric key
openssl rand -base64 32 > /srv/gitlab/tmp/symmetric_keyfile.key

# encrypt symmetric key with asm-key
openssl rsautl -encrypt -inkey /srv/gitlab/public_key.pem -pubin -in /srv/gitlab/tmp/symmetric_keyfile.key -out /srv/gitlab/tmp/symmetric_keyfile.enc

# generate hashes
sha1sum $BACKUPCONFIG >> /srv/gitlab/tmp/$(basename $BACKUPCONFIG).sha1sum
sha1sum $BACKUPDATA >> /srv/gitlab/tmp/$(basename $BACKUPDATA).sha1sum

# encrypt backup files
openssl enc -in $BACKUPCONFIG -out /srv/gitlab/tmp/$(basename $BACKUPCONFIG).enc -e -aes256 -kfile /srv/gitlab/tmp/symmetric_keyfile.key
openssl enc -in $BACKUPDATA -out /srv/gitlab/tmp/$(basename $BACKUPDATA).enc -e -aes256 -kfile /srv/gitlab/tmp/symmetric_keyfile.key

# archive backup files
cd /srv/gitlab/tmp
tar -cvzf /backup/gitlab-backup-$(date +"%Y-%m-%d-%H-%M").tgz *.enc *.sha1sum 1>/dev/null

# remove tmp files
rm -rf /srv/gitlab/tmp

# delete old config backups
find /srv/gitlab/backup-config -type f -mtime +7 -name '*config_gitlab_backup.tgz' -delete
find /srv/gitlab/backup-data -type f -mtime +7 -name '*data_gitlab_backup.tar' -delete
find /backup -type f -mtime +30 -name 'gitlab-backup-*.tgz' -delete 

Splunk Version 9.4.4 shows error while starting - VM CPU Flags are missing

Problem 

When you update your Splunk to e.g. version 9.4.4 and get this error while starting splunk:

Migrating to:
VERSION=9.4.4
BUILD=f627d88b766b
PRODUCT=splunk
PLATFORM=Linux-x86_64

********** BEGIN PREVIEW OF CONFIGURATION FILE MIGRATION **********

-> Currently configured KVSTore database path="/opt/splunk/var/lib/splunk/kvstore"
CPU Vendor: GenuineIntel
CPU Family: 15
CPU Model: 6
CPU Brand: \x
AVX Support: No
SSE4.2 Support: No
AES-NI Support: No

-> isSupportedArchitecture=0
-> isKVstoreDisabled=0
-> isKVstoreDatabaseFolderExist=0
-> isKVstoreDiagnosticsFolderExist=0
-> isKVstoreVersionFileFolderExist=1
-> isKVstoreVersionFileFolderEmpty=0
-> isKVstoreVersionFileMatched=1
-> isKVstoreVersionFromBsonMatched=0
-> isSupportedArchitecture=0
* Active KVStore version upgrade precheck FAILED!
  -- This check is to ensure that KVStore version 4.2 been in use.
  -- In order to fix this failed check, re-install the previous Splunk version, and follow the KVStore upgrade documentation: https://docs.splunk.com/Documentation/Splunk/9.3.0/Admin/MigrateKVstore#Upgrade_KV_store_server_to_version_4.2 .
Some upgrade prechecks failed!
ERROR while running splunk-preinstall.
 

Cause

This might be related to missing CPU features AVX, SSE4.2 and AES-NI to the Splunk VM, which are necessary for the new kvstore mongodb version, which is introduced in Splunk version 9.4: https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/administer-the-app-key-value-store/upgrade-the-kv-store-server-version#Upgrade_the_KV_store_server_version

You can check inside your VM using:
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$ grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u
splnonroot@devubu22h102:/opt/splunk$



Solution 

In your VM hypervisor (VMware ESXi, Microsoft Hyper-V, Proxmox, etc..) --> give the Splunk VMs the necessary CPU flags/features.

Example for proxmox:

1. Check inside your VM: grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u
2. Edit /etc/pve/qemu-server/*Your VM ID*.conf and add CPU: Host - so all the Host CPU Hardware flags are forwarded to the VM
3. Reboot the VM 
4. Check inside your VM again: grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u
5. Start Splunk 

Before Proxmox VM has CPU features: 

root@proxmox1:~#
root@proxmox1:~#
root@proxmox1:~# cat /etc/pve/qemu-server/*Your VM ID*.conf
boot: order=scsi0;ide2;net0
cores: 2
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]

[snapshot-pre-splunkupdate]
boot: order=scsi0;ide2;net0
cores: 2
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]
root@proxmox1:~#
root@proxmox1:~# 

After Proxmox VM has CPU features:

1.
root@proxmox1:~#
root@proxmox1:~#
root@proxmox1:~# cat /etc/pve/qemu-server/*Your VM ID*.conf
boot: order=scsi0;ide2;net0
cores: 2
cpu: host
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]

[snapshot-pre-splunkupdate]
boot: order=scsi0;ide2;net0
cores: 2
cpu: host
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]
root@proxmox1:~#
root@proxmox1:~#

2. Reboot the VM

3. Then inside your VM the CPU flags are visible:
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$ grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u
aes
avx
sse4_2
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$ 

4. Start Splunk again

Update Nginx ProxyManager docker container guide

Commands

1. Backup your container
2. Check which version your Nginx ProxyManager is currently running by:
   docker exec -it nginx_app_1 /bin/bash


3. Check which docker containers are currently running
   docker ps
   
   
4. Stop the Nginx ProxyManager application and database containers by:
   docker stop nginx_app_1
docker stop nginx_db_1
   
   
5. Pull the latest (or a specific) version of the image by:
   docker pull jc21/nginx-proxy-manager:latest
   
   
6. Start the Containers
   docker-compose -f nginx.yml up -d
   
   
7. Check the logs of the containers
   docker logs --follow nginx_app_1


8. Check which version your Nginx ProxyManager is currently running by:
   docker exec -it nginx_app_1 /bin/bash


9. Check your monitoring solution & test your applications




Example

user@container-nginx:~#
user@container-nginx:~# docker ps
CONTAINER ID   IMAGE                      COMMAND             CREATED         STATUS         PORTS                                                                                  NAMES
1a74a14bc3ab   9c3f57826a5d               "/init"             18 days ago   Up 4 minutes   0.0.0.0:80-81->80-81/tcp, :::80-81->80-81/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   nginx_app_1
02372069f98d   jc21/mariadb-aria:latest   "/scripts/run.sh"   18 days ago   Up 4 minutes   3306/tcp                                                                               nginx_db_1
user@container-nginx:~#
user@container-nginx:~#
user@container-nginx:~# docker stop nginx_app_1
nginx_app_1
user@container-nginx:~# docker stop nginx_db_1
nginx_db_1
user@container-nginx:~#
user@container-nginx:~# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
user@container-nginx:~#
user@container-nginx:~# docker pull jc21/nginx-proxy-manager:latest
latest: Pulling from jc21/nginx-proxy-manager
7cf63256a31a: Pull complete
191fb0319d69: Pull complete
9ace5189354c: Pull complete
e4db5efc926a: Pull complete
[...]
be35f3c3bf02: Pull complete
Digest: sha256:e5eecad9bf040f1e7ddc9db6bbc812d690503aa119005e3aa0c24803746b49ea
Status: Downloaded newer image for jc21/nginx-proxy-manager:latest
docker.io/jc21/nginx-proxy-manager:latest
user@container-nginx:~#
user@container-nginx:~#
user@container-nginx:~# ls -lah
total 676K
[...]
-rw-r--r--  1 user    user    607K May 14 02:55 cron-auto-update.log
drwxr-xr-x  7 user    user    4.0K Nov  5  2023 data
drwxr-xr-x  8 user    user    4.0K May 14 19:07 letsencrypt
drwxr-xr-x  5 postfix crontab 4.0K May 14 19:14 mysql
-rw-r--r--  1 user    user    1.1K Aug 11  2024 nginx.yml
user@container-nginx:~#
user@container-nginx:~#
user@container-nginx:~#
user@container-nginx:~#
user@container-nginx:~# docker-compose -f nginx.yml up -d
Starting nginx_db_1 ... done
Recreating nginx_app_1 ... done
user@container-nginx:~#
user@container-nginx:~#
user@container-nginx:~# docker exec -it nginx_app_1 /bin/bash
 _   _       _            ____                      __  __
| \ | | __ _(_)_ __ __  _|  _ \ _ __ _____  ___   _|  \/  | __ _ _ __   __ _  __ _  ___ _ __
|  \| |/ _` | | '_ \\ \/ / |_) | '__/ _ \ \/ / | | | |\/| |/ _` | '_ \ / _` |/ _` |/ _ \ '__|
| |\  | (_| | | | | |>  <|  __/| | | (_) >  <| |_| | |  | | (_| | | | | (_| | (_| |  __/ |
|_| \_|\__, |_|_| |_/_/\_\_|   |_|  \___/_/\_\\__, |_|  |_|\__,_|_| |_|\__,_|\__, |\___|_|
       |___/                                  |___/                          |___/
Version 2.12.3 (c5a319c) 2025-03-12 00:21:07 UTC, OpenResty 1.27.1.1, debian 12 (bookworm), Certbot certbot 3.2.0
Base: debian:bookworm-slim, linux/amd64
Certbot: nginxproxymanager/nginx-full:latest, linux/amd64
Node: nginxproxymanager/nginx-full:certbot, linux/amd64

[yp@docker-9a056abb3b01:/app]#

 

This also works fine if the docker container is within an LXC container. It should also work fine with podman instead of docker.

Splunk SearchHead Cluster Artifact Proxying - Splunk internally sharing cached search results

When the same search is run twice in a splunk cluster, is it using a cache for the results or searching the data a second time?

A splunk search head search artifact means the results and metadata from a completed splunk search job (see: https://docs.splunk.com/Splexicon:Searchartifact)

So an artifact is a complete search which is cached for 10minutes.

In a search head cluster the search artifacts are replicated. However this takes a few seconds. What happens, if a search is run again on another search head and the artifact isnt replicated yet to that search head?

The search head captain, which streers those requests uses artifact proxying so the artifact is proxied from the search head which already has completed the search to the other search head.

See also: https://docs.splunk.com/Documentation/Splunk/9.4.0/DistSearch/SHCarchitecture#How_the_cluster_handles_search_artifacts

Example

• 15 May 2025 11:21:01am - User1 starts the search "index=abc sourcetype=def" @04.May 2025 05:00:00am to 06:00:00am on SH1
• 15 May 2025 11:21:03am - The search "index=abc sourcetype=def" @04.May 2025 05:00:00am to 06:00:00am on SH1 is complete
• 15 May 2025 11:21:13am - User2 searches "index=abc sourcetype=def" @04.May 2025 05:00:00am to 06:00:00am on SH2
• The search head cluster captain will proxy the search artifact (search results) from SH2 to SH1, so the search mustnt run a second time

SPL query for Splunk proxied artifacts

index=_internal host IN (searchhead01*,searchhead02*,searchhead03*) sourcetype=splunkd_access uri_path="/services/search/jobs*" isProxyRequest=true | stats count by method host file

Splunk UseCase for attacks against FortiGate Firewall management interfaces

If you are using Splunk as your SIEM you can try to detect attacks against your FortiGate firewalls by using the following SPL query:

index=firewall type=event subtype=system msg IN ("Failed to match community*", "Message authentication or checking failed*", "Negotiation failed: no matching*", "Negotiation failed: Broken pipe*")
| stats earliest(_time) as FirstEvent count by devname,msg,result,logdesc
| eval FirstEvent=strftime(FirstEvent,"%Y-%m-%d %H:%M:%S")


Additionally: It is imperative that you protect your FortiGate interfaces with TrustedHosts AND Local-In-Policies. Only using TrustHosts protects HTTPS, SSH, etc but not other protocols like SIP, IPsec, CAPWAP, BGP, OSPF, SSLVPN etc which are also local services running on the FortiGate, which need to be protected, too.
See https://how2itsec.blogspot.com/2022/10/fortigate-admin-interface.html

Splunk alert for buckets which are not correctly replicated

The following shows a splunk savedsearch/alert which searches for Splunk buckets which are not correctly replicated to all indexers. 

Example

For example if you have a multisite cluster having 2 sites and each site should contain 2 copies of a bucket: 

splunk_server_clustering_available_sites: "site1,site2"
splunk_server_clustering_site_replication_factor: 'origin:1, site1:2, site2:2, total:4'
splunk_server_clustering_site_search_factor: 'origin:1, site1:2, site2:2, total:4'

Then the following SPL or savedsearch/alert might help identify if multiple buckets of an index are only replicated once:

| dbinspect index=* ```<-- show all buckets of all indexes ``` 
|search NOT state=hot ```<-- only warm & cold buckets ``` 
|eventstats count by bucketId  ```<-- list all bucket-ids only once, count how often they occur ``` 
|search count<2 ```<-- filter for all buckets that occur only once and are not replicated 4 times ``` 
|stats count by index ```<-- show all indexes that have buckets which were replicated only once ``` 
|search count>10 ```<-- show all indexes that have more than 10 buckets which were replicated only once```
``` All buckets should be replicated 4 times according to the search/replication factor of the Splunk multisite cluster. This alert shows if there are indexes with over 10 buckets that are only present once instead of being replicated on 4 indexers``` 

Screenshot:

Explaining screenshot:

How to bring relatives & friends to use a password manager

After about two years of effort, here's my personal experience report on "How do I get my relatives/friends to use a password manager?"

• It helps to use a free product (e.g., KeePass, Bitwarden, etc.), so there is no price argument against it
  
• It helps to use a widely audited open-source product (e.g., KeePass, Bitwarden, etc.), to establish trust in the provider/solution
  
• It's important to convince relatives/friends with convenience. Unfortunately, most people are not very interested in security. Convenience motivates long-term use. Prerequisites for convenience include:
• Access to the same vault from PC, tablet, and phone (usually via vault-in-cloud)
• Auto-Type for automatic password entry on all devices (PC/laptop & tablet/phone)
• Detection of new/changed manual typed in credentials with one-click saving to the password manager.
• Support for passkeys/cert-based-auth for automatic login
• Biometric or simple (but secure) unlocking of the password manager (personally, I only trust devices with an audited secure enclave chip for storing biometric data + a biometric scanner which isnt bypassed by simply showing a picture + protection against brute force/abuse + remote-wipe/remote-lock capability; but many people already use their fingerprint/faceprint on their phone, which can then be used).

3 Phases

I have had the best experiences with a migration/integration divided into three phases:

1. Force people to use it - it's important to set up the password manager on their PC, tablet, and phone, with automatic password entry and detection when new credentials are manually entered, so they can be saved directly in the password manager.
2. Collect all accounts over a few weeks/months and convince people of the convenience (automatic login/auto-type) of the password manager. (A bit like the initial discovery phase of an asset inventory or a CMDB)
3. By now, people don't care about the password anymore. So, I can introduce complex, unique passwords. This starts especially with email accounts, etc.
4. (The fourth phase would be to set up MFA everywhere, possibly integrating it into the password manager if possible, and changing all passwords to unique & strong ones.)

Notes

a) Depending on the age or willingness to try new things, I also set up MFA for email accounts right in Phase 1. For some, setting up a password manager and MFA at the same time leads to overwhelm, so take it slowly step by step, first the password manager, then MFA a few weeks later. Besides, MFA is great, but for email, one factor (username+password) usually still works via POP3/IMAP/SMTP/ActiveSync, so a strong password is still important (email accounts are among the most important accounts because of the password reset option of all the accounts).

b) It's worth enabling the automatic check on haveibeenpwnd and similar services in a password manager if such a feature is available. Often, it reveals that passwords, which have never been changed, have been compromised for years.

c) A password manager also needs to be hardened, e.g., by increasing KDF iterations, etc.

d) Have a backup from the beginning (e.g., Recovery Key, Master Password on paper in a safe/folder, in case it is forgotten, etc.). This ensures that if people make a mistake, you still have access to your password manager vault. For example, I once had a case where a user accidentally deleted their MFA entry. I had set up a backup for the MFA app (it was Microsoft Authenticator in this case), but the backup only protects against device loss, not manual deletion of entries. So, we needed a recovery key, which I had also set up. So, think about misuse and have a backup for it.

e) For IT/IT-Sec folks, I recommend giving them YubiKeys. Just gift them one or two (backup) and force them to use them for their own good :-) It quickly takes off once the first account is set up.

f) This takes time. For example, last week, I visited close friends (both mid-30s) for dinner, so I could push Phase 3 on them. It took another 2 hours, but all questions were answered, and they now love using Bitwarden and recommend it themselves. But you need the time. Confused users won't use it, which would be sad 😊.

Using this method, I have now got a small double-digit number of people to use Bitwarden permanently. And they, in turn, have started getting others to use it. Many small steps. I think it's great ☺️.

 

German / Deutsch:

Nach ca 2Jahren Bemühungen mein persönlicher  Erfahrungsbericht zu „Wie kriege ich meine Verwandten/Freunde dazu einen Passwort-Manager zu verwenden?“

• Hilfreich ist es auf einen kostenloses Produkt zu setzen (zB KeePass, Bitwarden,..), dann kommt kein Preis Gegenargument
• Hilfreich ist es auf ein mehrfach auditiertes OpenSource Produkt zu setzen (zB KeePass, Bitwarden,..), damit Vertrauen zum Anbieter/zur Lösung hergestellt werden kann
• Wichtig ist, dass man die Verwandten/Freunde mit Komfort überzeugt. Sicherheit interessiert leider die meisten wenig/nicht genug. Komfort motiviert, dass es langfristig genutzt wird. Für Komfort ist Voraussetzung:
• Zugriff von PC, Tablet und Handy auf ein und den selben Vault (i.d.R. via Vault-in-Cloud)
• Auto-Type für automatische Eingabe der Passwörter auf allen Geräten (PC/Notebook & Tablet/Handy)
• Erkennung von neuen/geänderten manuellen Credentials mit-1-Klick-Speichern-in-PW-Manager
• Support von Pass-Keys/ Zert-Based-Auth für automatische Anmeldung
• Biometrische bzw einfache (aber sichere) Entsperrung des Passwort-Managers (wobei ich persönlich da nur bei Geräten mit auditierter SecureEnclave als Speicherort für Biometrische Daten vertraue + wenn nicht nur ein normales Bild zum entsperren reicht + Schutz vor BruteForce/Abuse + RemoteWipe/RemoteLock; Aber viele nutzen ja auf ihrem Handy bereits ihren Fingerprint/Faceprint, das kann dann mitgenutzt werden)

3 Phasen

Ich habe die besten Erfahrungen gemacht durch eine Migration/Integration, die ich in drei Phasen eingeteilt habe:

1. Zuerst mal die Leute überhaupt zwingen - wichtig ist, den PW-Manager auf deren PC, Tablet und Handy einzurichten, mit automatischer PW-Eingabe sowie Erkennung, wenn neue Credentials irgendwo manuell eingegeben werden, dass man diese direkt im PW-Manager speichern kann
2. Dadurch ein paar Wochen/Monate alle Accounts sammeln und die Leute vom Komfort (automatische Anmeldung/Auto-Type) das Passwort-Managers überzeugen. (Ein bisschen wie die erste Discovery-Phase eins Asset-Inventorys beziehungsweise einer CMDB)
3. Den Leuten ist mittlerweile das Passwort egal geworden. Also kann ich auch komplexe unique Passwörter einführen. Da beginnt man vor allem bei den Mailaccounts, usw.
4. (Die vierte Phase wäre dann überall MFA einzurichten, gegebenenfalls diesen auch in den Passwort-Manager mit zu integrieren, falls das möglich ist, sowie alle Passwörter auf unique & starke PWs zu ändern)

Anmerkungen

a) Je nach Alter bzw Bereitschaft-auf-Neues habe ich ihn Phase1 auch gleich MFA für die Mail-Accounts mit eingerichtet. Bei manchen führt PW-Manager & MFA gleichzeitig einrichten zur Überforderung, da dann iterativ Stück für Stück langsam vorgehen, und erst den PW-Manager, ein paar Wochen später MFA. Außerdem ist MFA super, aber bei Mail geht ja meistens trotzdem ein Faktor (Username+Password) via POP3/IMAP/SMTP/ActiveSync weiterhin, d.h. da ist ein starkes PW weiterhin wichtig (Mail-Accounts gehören zu den wichtigsten Accounts gehören, wegen der Passwort-Reset-Option für Accounts)

b) Es lohnt sich ein Passwort Manager gleich die Funktion automatischer Abgleich auf haveibeenpwnd und Co zu aktivieren, falls es eine solche Funktion gibt. Da kommt oft genug raus, dass nie geänderte Passwörter seit Jahren kompromittiert sind

c) Auch ein PW-Manager muss gehärtet werden, dh zB KDF-Iterations hochsetzen, usw.

d) Von Anfang an überall ein Backup haben (zb Recovery Key, Master-PW auf Zettel in Tresor/Ordner, falls es vergessen wird,.. Notfallzugriff eben), so dass man sich sicher sein kann, dass wenn die Leute etwas falsch machen,  man trotzdem noch Zugriff auf seinen Passwort-Manager Vault hat. ZB hatte ich zu Beginn einmal den Fall, dass ein Benutzer versehentlich dein MFA Eintrag gelöscht hat. Ich hatte extra dazu ein Backup für die MFA-App (war Microsoft Authenticator in dem Fall) eingerichtet, das Backup schützt aber nur bei Geräteverlust, nicht bei manuellen Löschen von Einträgen. Da haben wir also einen Recovery-Key gebraucht, den ich auch eingerichtet hatte. Also auch an Fehlbenutzung denken und ein Backup dafür haben.

e) Für IT/IT-Sec‘ler empfehle ich ihnen YubiKeys aufzudrücken. Einfach einen bzw zwei (Backup) schenken und sie zu ihrem Glück zwingen :-) Startet dann schnell von alleine, wenn der erste Account eingerichtet ist.

f) Das kostet Zeit. ZB war ich letzte Woche bei einem gut befreundeten Paar (ca Mitte 30) zum Abendessen, damit ich ihnen danach noch Phase3 aufs Auge drücken durfte/wollte. Hat wieder 2 Stunden gekostet, aber so waren alle Fragen geklärt und die beiden benutzen Bitwarden mittlerweile total gern und empfehlen es selber. Aber die Zeit braucht man. Verwirrte Benutzer nutzen es sonst nicht, wäre ja schade drum 😊

Ich habe mit dieser Methode mittlerweile eine kleine zweistellige Anzahl an Leuten dauerhaft zu Bitwarden gekriegt. Und die wiederum haben angefangen andere darauf zu kriegen. Viele kleine Schritte. Find ich gut ☺️

Windows Persistence Map v0.1

Mitres Att&ck framework writes about persistence TA0003: "The adversary is trying to maintain their foothold." 

There are multiple ways to get persistence in a Microsoft Windows operating system. Pepe Berba has created a nice overview of linux persistence techniques as a map, so I tried to do the same thing for Windows. This is version v0.1 of it:

• https://github.com/flostyen/windows-persistence
• SVG https://github.com/flostyen/windows-persistence/blob/main/Windows%20Persistence%20Map%20v0.1.svg 
• PDF https://github.com/flostyen/windows-persistence/blob/main/Windows%20Persistence%20Map%20v0.1.pdf
  

 

 

Edge browser internal debug tools - example network traffic

 Microsoft Edge browser has some internal tools: edge://edge-urls/

Example usage net-export - debugging network traffic

1. edge://net-export/ 
   
   
   
   
   
2. Start & set file location (edge-net-export-log.json)
3. Reproduce the issue in a new tab
4. Stop recording
5. The recording can be viewed via: https://netlog-viewer.appspot.com/#import
   → See Privacy: https://chromium.googlesource.com/catapult/+/master/netlog_viewer/
   "This app loads NetLog files generated by Chromium's chrome://net-export. Log data is processed and visualized entirely on the client side (your browser). Data is never uploaded to a remote endpoint."
6. Select and load the file
7. For example, Proxy/ProxyPAC Configuration: https://netlog-viewer.appspot.com/#proxy
   
   
   
   
   
8. For example, detailed Event Timeline: https://netlog-viewer.appspot.com/#events
   
   
   
   
   
9. For example, detailed DNS Events:
   
   
   
   
   
10. Detailed Socket overview: https://netlog-viewer.appspot.com/#sockets
11. Detailed HTTP/2 overview: https://netlog-viewer.appspot.com/#http2
12. Detailed QUIC overview: https://netlog-viewer.appspot.com/#quic
    
    

Update Nextron Aurora lite EDR Agent

To manually update Nextrons Aurora Lite EDR agent, follow the steps: https://aurora-agent-manual.nextron-systems.com/en/latest/usage/upgrade-and-updates.html

1. Download Aurora Lite files & license: https://www.nextron-systems.com/aurora/
2. Unzip the files into a folder
3. Copy the license file into that folder
4. Start a PowerShell with Admin rights
5. Execute aurora-agent-util.exe upgrade --restart-service

Example

PS C:\Users\clw11c493\Downloads\aurora-agent-lite-win-pack_v1.2.1>
PS C:\Users\clw11c493\Downloads\aurora-agent-lite-win-pack_v1.2.1> aurora-agent-util.exe upgrade --restart-service
Aug 10 19:30:37 clw11c493 AURORA: Info MODULE: Aurora-Agent MESSAGE: License file found OWNER: some@address.com VALID: true VALID_FROM: 2024/04/15 VALID_TO: 2025/02/21
Aug 10 19:30:37 clw11c493 AURORA: Info MODULE: Aurora-Agent MESSAGE: Checking for new version PRODUCT: aurora-agent-lite-win
Aug 10 19:31:08 clw11c493 AURORA: Info MODULE: Aurora-Agent MESSAGE: Stopped installed Aurora Agent service
Aug 10 19:31:08 clw11c493 AURORA: Info MODULE: Aurora-Agent MESSAGE: Installing downloaded package INSTALL_PATH: C:\Program Files\Aurora-Agent
Aug 10 19:31:13 clw11c493 AURORA: Info MODULE: Aurora-Agent MESSAGE: Started installed Aurora Agent service
Aug 10 19:31:13 clw11c493 AURORA: Info MODULE: Aurora-Agent MESSAGE: Updated Aurora Agent NEW: 1.2.1 OLD: 1.1.5
PS C:\Users\clw11c493\Downloads\aurora-agent-lite-win-pack_v1.2.1>
PS C:\Users\clw11c493\Downloads\aurora-agent-lite-win-pack_v1.2.1>  

 

To debug aurora you can use aurora-agent-64.exe --debug

Nextron Aurora EDR agent shows \Pr Error

Problem

During start of Nextrons Aurora EDR lite agent the programm shows the following error message:

PS C:\Program Files\Aurora-Agent> aurora-agent-64.exe --dashboard
      ___                                  __    _ __
     /   | __  ___________  _________ _   / /   (_) /____
    / /| |/ / / / ___/ __ \/ ___/ __ `/  / /   / / __/ _ \
   / ___ / /_/ / /  / /_/ / /  / /_/ /  / /___/ / /_/  __/
  /_/  |_\__,_/_/   \____/_/   \__,_/  /_____/_/\__/\___/


  Aurora Agent Lite Version 1.2.1 (9da9fbf29275c), Signature Revision 2024/08/10-134221 (Sigma r2024-07-17-29-gace902b68)
  (C) Nextron Systems GmbH, 2022

Aug 10 19:51:16 clw11c493 AURORA: Error MODULE: EventDistributor MESSAGE: Could not parse process exclude ERROR: error parsing regexp: invalid character class range: `\Pr` LINE: error parsing regexp: invalid character class range: `\Pr`
Aug 10 19:51:16 clw11c493 AURORA: Error MODULE: EventDistributor MESSAGE: Could not parse process exclude ERROR: error parsing regexp: invalid character class range: `\Pr` LINE: error parsing regexp: invalid character class range: `\Pr`
Aug 10 19:51:16 clw11c493 AURORA: Error MODULE: EventDistributor MESSAGE: Could not parse process exclude ERROR: error parsing regexp: invalid character class range: `\Pr` LINE: error parsing regexp: invalid character class range: `\Pr`

Solution

Your "process-excludes.cfg" (C:\Program Files\Aurora-Agent\config\process-excludes.cfg) configurations probably has a missing escaping "\" in the process-path (aurora searches for those process paths using regular expression):

Wrong:

^"C:\Program Files (x86)\

Correct:

^"C:\\Program Files (x86)\\

 

Azure Managed Identities (technical service accounts)

Explaination

• Azure Managed Identities = technical service accounts
• Password is automatically managed, as it was the case in managed service accounts in OnPrem ActiveDirectory
• Managed Identity types:
• System Managed Idendity ==> strictly assigned to a single Azure system (like a VM), cant be shared with another system
• User Managed Identity ==> for example for HA-clusters, in which all HA-nodes need the same user

Managed Identities dont use a password, instead they use OAuth2 and its token --> https://169.254.169.254/metadata/identity/oauth2/token 

Source: https://medium.com/@siddiquimohammad0807/azure-managed-identity-types-and-importance-c64f6292577d

Example

• When using PowerShell Connect-AzAccount --> a new window for username+password+mfa is opened --> these will be used as credentials
  
  
  
• When using PowerShell Connect-AzAccount - Identity --> no new window is opened, instead the managed identity is used --> Powershell sends HTTPS OAuth2 Query to Azure IMDS "Instance Meta Data Service" and received a token, which is then used
  

Windows PowerShell
Install the latest PowerShell for new features and improvements! https://aka.ms/PSWindows
PS C:\Users\Administrator20>
PS C:\Users\Administrator20>
PS C:\Users\Administrator20> Install-Module Az 
NuGet provider is required to continue 
PowerShellGet requires NuGet provider version '2.8.5.201' or newer to interact with NuGet-based repositories. The NuGetprovider must be available in 'C:\Program Files\PackageManagement\ProviderAssemblies'  or 'C:\Users\Administrator20\AppData\Local\PackageManagement\ProviderAssemblies'. You can also install the NuGet provider by running 'Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force'. Do you want PowerShellGet to install
 and import the NuGet provider now?
[Y] Yes[N] No[S] Suspend[?] Help (default is "Y"): y
 
Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its
InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install the modules from
'PSGallery'?
[Y] Yes[A] Yes to All[N] No[L] No to All[S] Suspend[?] Help (default is "N"): y
PS C:\Users\Administrator20>
PS C:\Users\Administrator20>
PS C:\Users\Administrator20> Connect-AzAccount -Identity
 
Subscription name Tenant
----------------- ------
274102ec-cd24-4af2-a4c2-832941ce526f
 
 
PS C:\Users\Administrator20>
PS C:\Users\Administrator20>




Cribl - Change values to lowerCase

Some logs (e.g. Microsoft Azure) sometimes are not fully normalized to all lowercase characters. You can use Cribl to adjust those values by either use the eval or mask function:

Eval

_raw.toLowerCase()

https://docs.cribl.io/stream/eval-function/ 

"The Eval Function adds or removes fields from events. (In Splunk, these are index-time fields.)"

Mask

You can also use Cribls mask function to hit all fields:

Regex = (.*)        <---- 1st Capturing Group (.*), see https://regex101.com/

g1.toLowerCase

https://docs.cribl.io/stream/mask-function/

"The Mask Function masks, or replaces, patterns in events. This is especially useful for redacting PII (personally identifiable information) and other sensitive data."

Filter logs in Splunk - example filtering monitor probe checks

When running Splunk you want to filter logs, for example to get rid of the many health check probe querys from your monitoring system.

Example filtering PRTG monitoring probe requests using props.conf and transforms.conf

1. Find the monitoring probes in the logs in splunk, e.g.:

10.148.227.111 - - [18/Jul/2024:23:21:06 +0200] "GET /login HTTP/1.1" 200 12882 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:21:06 +0200] "GET / HTTP/1.1" 302 5793 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:20:56 +0200] "GET /login HTTP/1.1" 200 12882 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:20:56 +0200] "GET / HTTP/1.1" 302 5790 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.121 - - [18/Jul/2024:23:12:17 +0200] "GET /login HTTP/1.1" 200 17480 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com; 576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:16 +0200] "GET / HTTP/1.1" 302 5572 "-" Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com; 576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:15 +0200] "GET /login HTTP/1.1" 200 17486 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com; 576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:15 +0200] "GET /login HTTP/1.1" 200 17474 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com; 576bb8887fe66b1eece876e62e701b9e)"

2. Create a regex, which finds the logs (which a precise match but as less cpu steps as possible) using https://regex101.com/

In this example the following regexes where used:

Mozilla\/\d+\.\d+\s+\(compatible;\s+PRTG\s+Network\s+Monitor
Mozilla\/\d.\d\s\(compatible\;\sPaesslerCloudBot\/\d.\d
 

3. Create a dedicated splunk app for this log source or use the default splunk search app and modify the props.conf. Create an entry which you map to the host, source or sourcetype and tell it to use transforms.conf:

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#
uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local# cat props.conf
[...]

#filter prtg monitoring logs
[host::fqdn.of.logsource]
TRANSFORMS-t1=filter-prtg-from-access
TRANSFORMS-t2=filter-prtgcloud-from-access

4. Modify the transforms.conf of this same splunk app. Create an entry which you map to the host, source or sourcetype and force it to the nullQueue:

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#
uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local# cat transforms.conf
#filter prtg logs von access.log von nextcloud
#
[filter-prtg-from-access]
REGEX = Mozilla\/\d.\d\s\(compatible\;\sPRTG\sNetwork\sMonitor
DEST_KEY = queue
FORMAT = nullQueue

[filter-prtgcloud-from-access]
REGEX = Mozilla\/\d.\d\s\(compatible\;\sPaesslerCloudBot\/\d.\d
DEST_KEY = queue
FORMAT = nullQueue

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#

5. Reload the splunk configuration using https://your.splunk.fqdn:8000/en-GB/debug/refresh 

6. Your logs should be filtered. If not, check the btool to see if another splunk configuration takes precedence to your configuration:

./splunk btool props list
./splunk btool props list --debug
./splunk btool transforms list
./splunk btool transforms list --debug