Filter logs in Splunk - example filtering monitor probe checks

When running Splunk you want to filter logs, for example to get rid of the many health check probe querys from your monitoring system.

Example filtering PRTG monitoring probe requests using props.conf and transforms.conf

1. Find the monitoring probes in the logs in splunk, e.g.:

10.148.227.111 - - [18/Jul/2024:23:21:06 +0200] "GET /login HTTP/1.1" 200 12882 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:21:06 +0200] "GET / HTTP/1.1" 302 5793 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:20:56 +0200] "GET /login HTTP/1.1" 200 12882 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:20:56 +0200] "GET / HTTP/1.1" 302 5790 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.121 - - [18/Jul/2024:23:12:17 +0200] "GET /login HTTP/1.1" 200 17480 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com; 576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:16 +0200] "GET / HTTP/1.1" 302 5572 "-" Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com;
576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:15 +0200] "GET /login HTTP/1.1" 200 17486 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com;
576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:15 +0200] "GET /login HTTP/1.1" 200 17474 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com;
576bb8887fe66b1eece876e62e701b9e)"

2. Create a regex, which finds the logs (which a precise match but as less cpu steps as possible) using https://regex101.com/

regex101.com regex splunk filter

In this example the following regexes where used:

Mozilla\/\d+\.\d+\s+\(compatible;\s+PRTG\s+Network\s+Monitor
Mozilla\/\d.\d\s\(compatible\;\sPaesslerCloudBot\/\d.\d

 

3. Create a dedicated splunk app for this log source or use the default splunk search app and modify the props.conf. Create an entry which you map to the host, source or sourcetype and tell it to use transforms.conf:

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#
uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local# cat props.conf
[...]

#filter prtg monitoring logs
[host::fqdn.of.logsource]
TRANSFORMS-t1=filter-prtg-from-access
TRANSFORMS-t2=filter-prtgcloud-from-access

4. Modify the transforms.conf of this same splunk app. Create an entry which you map to the host, source or sourcetype and force it to the nullQueue:

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#
uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local# cat transforms.conf
#filter prtg logs von access.log von nextcloud
#
[filter-prtg-from-access]
REGEX = Mozilla\/\d.\d\s\(compatible\;\sPRTG\sNetwork\sMonitor
DEST_KEY = queue
FORMAT = nullQueue

[filter-prtgcloud-from-access]
REGEX = Mozilla\/\d.\d\s\(compatible\;\sPaesslerCloudBot\/\d.\d
DEST_KEY = queue
FORMAT = nullQueue

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#

5. Reload the splunk configuration using https://your.splunk.fqdn:8000/en-GB/debug/refresh 

6. Your logs should be filtered. If not, check the btool to see if another splunk configuration takes precedence to your configuration:

./splunk btool props list
./splunk btool props list --debug
./splunk btool transforms list
./splunk btool transforms list --debug

Nextcloud shows error "Data directory and your files are probably accessible from the Internet"

Starting Nextcloud v29 the error "Data directory and your files are probably accessible from the Internet".

Nextcloud error "Data directory and your files are probably accessible from the Internet"

 

Cause

root@prdanc2049:/var/www/nextcloud/config# pwd
/var/www/nextcloud/config
root@prdanc2049:/var/www/nextcloud/config#
root@prdanc2049:/var/www/nextcloud/config# cat config.php
<?php
$CONFIG = array (
  'passwordsalt' => 'Redacted',
  'secret' => 'Redacted',
  'trusted_domains' =>
  array (
    0 => 'localhost',
    1 => '10.68.127.123',
    2 => 'nextcloud',

    3 => 'mypublic.domain.com',
  ),
  'datadirectory' => '/mnt/ncdata',
  'dbtype' => 'pgsql',
[...]

Solution

Remove the ip addresses, "localhost" and "nextcloud" from the trusted_domains in /var/www/nextcloud/config/config.php

root@prdanc2049:/var/www/nextcloud/config# pwd
/var/www/nextcloud/config
root@prdanc2049:/var/www/nextcloud/config#
root@prdanc2049:/var/www/nextcloud/config# cat config.php
<?php
$CONFIG = array (
  'passwordsalt' => 'Redacted',
  'secret' => 'Redacted',
  'trusted_domains' =>
  array (
    0 => 'mypublic.domain.com',
  ),
  'datadirectory' => '/mnt/ncdata',
  'dbtype' => 'pgsql',
[...]


Information regarding trusted_domains in the config.php: https://docs.nextcloud.com/server/stable/admin_manual/configuration_server/config_sample_php_parameters.html#trusted-domains

Your list of trusted domains that users can log into. Specifying trusted domains prevents host header poisoning. Do not remove this, as it performs necessary security checks.

You can specify:

  • the exact hostname of your host or virtual host, e.g. demo.example.org.

  • the exact hostname with permitted port, e.g. demo.example.org:443. This disallows all other ports on this host

  • use * as a wildcard, e.g. ubos-raspberry-pi*.local will allow ubos-raspberry-pi.local and ubos-raspberry-pi-2.local

  • the IP address with or without permitted port, e.g. [2001:db8::1]:8080 Using TLS certificates where commonName=<IP address> is deprecated

Fix Nextcloud missing database indexes

 

Nextcloud security warning database indexes missing

Nextcloud adminsitration page shows the following waring:

The database is missing some indexes. Due to the fact that adding indexes on big tables could take some time they were not added automatically. By running "occ db:add-missing-indices" those missing indexes could be added manually while the instance keeps running. Once the indexes are added queries to those tables are usually much faster. Missing optional index "mail_messages_msgid_idx" in table "mail_messages". Missing optional index "fs_storage_path_prefix" in table "filecache".

Solution

Login to your Nextcloud system and use the command "sudo -u www-data php /var/www/nextcloud/occ db:add-missing-indices" to fix it.
 
root@nextcloud:~#
root@nextcloud:~# sudo -u www-data php /var/www/nextcloud/occ db:add-missing-indices
Adding additional mail_messages_msgid_idx index to the oc_mail_messages table, this can take some time...
oc_mail_messages table updated successfully.
Adding additional fs_storage_path_prefix index to the oc_filecache table, this can take some time...
oc_filecache table updated successfully.
root@nextcloud:~#


Nextcloud behind nginx reverse proxy error on iPhone and iPad

When publishing a nextcloud website using a nginx reverse proxy, you might get an error shown on Apple iOS iPhone and iPadOS iPads on all browsers - e.g. Safari or Chrome: ERR_CONNECTION_CLOSED

Solution

Add in the nginx reverse proxy configuration the following line:
proxy_hide_header Upgrade;
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header 

By default, nginx does not pass the header fields “Date”, “Server”, “X-Pad”, and “X-Accel-...” from the response of a proxied server to a client. The proxy_hide_header directive sets additional fields that will not be passed. If, on the contrary, the passing of fields needs to be permitted, the proxy_pass_header directive can be used.

Syntax: proxy_hide_header field;
Default:
Context: http, server, location

Nginx GUI configuration:

  1. Login to your Nginx Proxy Manager.
  2. Open the 3 dots settings menu of the NextCloud host and select “Edit”
  3. In the tab menu at the top of the window that has just opened select “Advanced” and insert the following in the “Custom Nginx Configuration” box:
    proxy_hide_header Upgrade;
  4. Click "save". 

 

Source: https://help.nextcloud.com/t/nextcloud-behind-nginx-proxy-manager-and-safari-ios-macos-no-access/142234/13

Certbot Cloudflare DNS Challenge error --dns-cloudflare-credentials

When certbot is showing you the error "certbot: error: unrecognized arguments: --dns-cloudflare-credentials" you might find the following helpful:

Problem

root@linlxc033:~#
root@linlxc033:~# certbot certonly --dns-cloudflare --dns-cloudflare-credentials /root/tls/certbot/cloudflare-api.key --preferred-challenges dns-01 -d "somedomain.com" -d "*.somedomain.com"
usage:
  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
it will attempt to use a webserver both for obtaining and installing the
certificate.
certbot: error: unrecognized arguments: --dns-cloudflare-credentials /root/tls/certbot/cloudflare-api.key
root@linlxc033:~#
root@linlxc033:~#

Solution

Install the required package. This is the solution if you have installed python3-certbot as an apt package:

sudo apt -y install python3-certbot-dns-cloudflare

This is the solution if you have installed certbot as a snap:

sudo snap set certbot trust-plugin-with-root=ok
sudo snap install certbot-dns-cloudflare

Example

root@linlxc033:~#
root@linlxc033:~# sudo apt -y install python3-certbot-dns-cloudflare
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
[...]
root@linlxc033:~#
root@linlxc033:~# sudo snap set certbot trust-plugin-with-root=ok
root@linlxc033:~# sudo snap install certbot-dns-cloudflare
certbot-dns-cloudflare 2.8.0 from Certbot Project (certbot-eff✓) installed
root@linlxc033:~#
root@linlxc033:~#
root@linlxc033:~# certbot certonly --dns-cloudflare --dns-cloudflare-credentials /root/tls/certbot/cloudflare-api.key --preferred-challenges dns-01 -d "somedomain.com" -d "*.somedomain.com"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for somedomain.com and *.somedomain.com   
Unsafe permissions on credentials configuration file: /root/tls/certbot/cloudflare-api.key
Waiting 10 seconds for DNS changes to propagate

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/somedomain.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/somedomain.com/privkey.pem
This certificate expires on 2024-08-04.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
root@linlxc033:~#
root@linlxc033:~#
 

Color highlight & timestamp your bash cli prompt

 To color highlight your bash cli simply edit /etc/bash.bashrc and add the following lines:


force_color_prompt=yes

    if [ "$LOGNAME" = root ] || [ "`id -u`" -eq 0 ] ; then
        PS1='\[\033[01;31m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[01;34m\]#\033[00m\] '
    else
        PS1='\u@\h:\w\$ '
    fi
 

Example:

Additionally you can show the timestamp by adding export PROMPT_COMMAND="echo -n \[\$(date +%H:%M:%S)\]\ " into your ~/.bashrc file:

New proxmox VM does not boot

When adding a new VM (in this example the nextcloud appliance VM from https://www.hanssonit.se/nextcloud-vm/) to an old version of proxmox like version 6 (debian 10), the VM might not boot and stay stuck showing Booting from Hard Disk ...

Booting from Hard Disk ...

Solution

 

  1. Update your proxmox system, e.g. from version 6 (debian 10 - "buster") to promox version 7 (debian 11 - "bullseye"), see https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0
  2. Import the VM again and start it


 

Update proxmox 6.4.x to 7.x

Updating a proxmox system from version 6.4.x to 7.x using https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0

Proxmox VE 6.x is based on Debian 10.x which is called “buster”.
Proxmox VE 7.x is based on Debian 11.x which is called “bullseye”.

  1. Make sure you have a backup of all VMs, Containers, Proxmox itself etc.
  2. Login via SSH/CLI
  3. Check your sources.list file, should look like this:

    cat /etc/apt/sources.list

    deb http://deb.debian.org/debian
    bullseye main contrib
    deb http://deb.debian.org/debian 
    bullseye-updates main contrib
    # security updates
    deb http://security.debian.org 
    bullseye/updates main contrib

    You may use sed -i 's/buster\/updates/bullseye-security/g;s/buster/bullseye/g' /etc/apt/sources.list to update "buster" to "bullseye".

  4. Check the enterprise repository:

    cat /etc/apt/sources.list.d/pve-enterprise.list

    When running Proxmox VE 7.x with No-Subscription use:

    deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

    When running Proxmox VE 7.x with a subscription use:

    deb https://enterprise.proxmox.com/debian/pve
    bullseye pve-enterprise

  5. Check Proxmox version using:

    pveversion -v


  6. Run the pve6to7 script

    root@prxmx024a:~# pve6to7
    = CHECKING VERSION INFORMATION FOR PVE PACKAGES =

    Checking for package updates..
    PASS: all packages uptodate

    Checking proxmox-ve package version..
    PASS: proxmox-ve package has version >= 6.4-1

    Checking running kernel version..
    PASS: expected running kernel '5.4.203-1-pve'.

    = CHECKING CLUSTER HEALTH/SETTINGS =

    SKIP: standalone node.

    = CHECKING HYPER-CONVERGED CEPH STATUS =

    SKIP: no hyper-converged ceph setup detected!

    = CHECKING CONFIGURED STORAGES =

    PASS: storage 'local' enabled and active.
    PASS: storage 'local-lvm' enabled and active.
    PASS: storage 'storageusbhdd01' enabled and active.

    = MISCELLANEOUS CHECKS =

    INFO: Checking common daemon services..
    PASS: systemd unit 'pveproxy.service' is in state 'active'
    PASS: systemd unit 'pvedaemon.service' is in state 'active'
    PASS: systemd unit 'pvestatd.service' is in state 'active'
    INFO: Checking for running guests..
    PASS: no running guest detected.
    INFO: Checking if the local node's hostname 'proxmox1' is resolvable..
    INFO: Checking if resolved IP is configured on local node..
    PASS: Resolved node IP '192.168.2.106' configured and active on single interface.
    INFO: Checking backup retention settings..
    INFO: storage 'local' - no backup retention settings defined - by default, PVE 7.x will no longer keep only the last backup, but all backups
    PASS: no problems found.
    INFO: checking CIFS credential location..
    PASS: no CIFS credentials at outdated location found.
    INFO: Checking custom roles for pool permissions..
    INFO: Checking node and guest description/note legnth..
    PASS: All node config descriptions fit in the new limit of 64 KiB
    PASS: All guest config descriptions fit in the new limit of 8 KiB
    INFO: Checking container configs for deprecated lxc.cgroup entries
    PASS: No legacy 'lxc.cgroup' keys found.
    INFO: Checking storage content type configuration..
    PASS: no problems found
    INFO: Checking if the suite for the Debian security repository is correct..
    INFO: Make sure to change the suite of the Debian security repository from 'buster/updates' to 'bullseye-security' - in /etc/apt/sources.list:6
    SKIP: NOTE: Expensive checks, like CT cgroupv2 compat, not performed without '--full' parameter

    = SUMMARY =

    TOTAL:    20
    PASSED:   17
    SKIPPED:  3
    WARNINGS: 0
    FAILURES: 0
    root@prxmx024a:~#
    root@prxmx024a:~#


  7. Run the pve6to7 script with the parameter -full

    root@prxmx024a:~#
    root@prxmx024a:~# pve6to7 --full
    = CHECKING VERSION INFORMATION FOR PVE PACKAGES =

    Checking for package updates..
    PASS: all packages uptodate

    Checking proxmox-ve package version..
    PASS: proxmox-ve package has version >= 6.4-1

    Checking running kernel version..
    PASS: expected running kernel '5.4.203-1-pve'.

    = CHECKING CLUSTER HEALTH/SETTINGS =

    SKIP: standalone node.

    = CHECKING HYPER-CONVERGED CEPH STATUS =

    SKIP: no hyper-converged ceph setup detected!

    = CHECKING CONFIGURED STORAGES =

    PASS: storage 'local' enabled and active.
    PASS: storage 'local-lvm' enabled and active.
    PASS: storage 'storageusbhdd01' enabled and active.

    = MISCELLANEOUS CHECKS =

    INFO: Checking common daemon services..
    PASS: systemd unit 'pveproxy.service' is in state 'active'
    PASS: systemd unit 'pvedaemon.service' is in state 'active'
    PASS: systemd unit 'pvestatd.service' is in state 'active'
    INFO: Checking for running guests..
    PASS: no running guest detected.
    INFO: Checking if the local node's hostname 'proxmox1' is resolvable..
    INFO: Checking if resolved IP is configured on local node..
    PASS: Resolved node IP '192.168.2.106' configured and active on single interface.
    INFO: Checking backup retention settings..
    INFO: storage 'local' - no backup retention settings defined - by default, PVE 7.x will no longer keep only the last backup, but all backups
    PASS: no problems found.
    INFO: checking CIFS credential location..
    PASS: no CIFS credentials at outdated location found.
    INFO: Checking custom roles for pool permissions..
    INFO: Checking node and guest description/note legnth..
    PASS: All node config descriptions fit in the new limit of 64 KiB
    PASS: All guest config descriptions fit in the new limit of 8 KiB
    INFO: Checking container configs for deprecated lxc.cgroup entries
    PASS: No legacy 'lxc.cgroup' keys found.
    INFO: Checking storage content type configuration..
    PASS: no problems found
    INFO: Checking if the suite for the Debian security repository is correct..
    INFO: Make sure to change the suite of the Debian security repository from 'buster/updates' to 'bullseye-security' - in /etc/apt/sources.list:6
    SKIP: No containers on node detected.

    = SUMMARY =

    TOTAL:    20
    PASSED:   17
    SKIPPED:  3
    WARNINGS: 0
    FAILURES: 0
    root@prxmx024a:~#

  8. Update your repository and packages:

    apt update

  9. Now upgrade the packages:

    apt dist-upgrade

  10. Reboot to activate the new Kernel, to check if you got all packages, run 'pveversion -v' and compare your output (all packages should have equal or higher version numbers): 
  11. Check Proxmox version using  

    pveversion -v



Example:

root@prxmx024a:~#
root@prxmx024a:~# pve6to7
= CHECKING VERSION INFORMATION FOR PVE PACKAGES =

Checking for package updates..
PASS: all packages uptodate

Checking proxmox-ve package version..
PASS: proxmox-ve package has version >= 6.4-1

Checking running kernel version..
PASS: expected running kernel '5.4.203-1-pve'.

= CHECKING CLUSTER HEALTH/SETTINGS =

SKIP: standalone node.

= CHECKING HYPER-CONVERGED CEPH STATUS =

SKIP: no hyper-converged ceph setup detected!

= CHECKING CONFIGURED STORAGES =

PASS: storage 'local' enabled and active.
PASS: storage 'local-lvm' enabled and active.
PASS: storage 'storageusbhdd01' enabled and active.

= MISCELLANEOUS CHECKS =

INFO: Checking common daemon services..
PASS: systemd unit 'pveproxy.service' is in state 'active'
PASS: systemd unit 'pvedaemon.service' is in state 'active'
PASS: systemd unit 'pvestatd.service' is in state 'active'
INFO: Checking for running guests..
PASS: no running guest detected.
INFO: Checking if the local node's hostname 'proxmox1' is resolvable..
INFO: Checking if resolved IP is configured on local node..
PASS: Resolved node IP '192.168.2.106' configured and active on single interface.
INFO: Checking backup retention settings..
INFO: storage 'local' - no backup retention settings defined - by default, PVE 7.x will no longer keep only the last backup, but all backups
PASS: no problems found.
INFO: checking CIFS credential location..
PASS: no CIFS credentials at outdated location found.
INFO: Checking custom roles for pool permissions..
INFO: Checking node and guest description/note legnth..
PASS: All node config descriptions fit in the new limit of 64 KiB
PASS: All guest config descriptions fit in the new limit of 8 KiB
INFO: Checking container configs for deprecated lxc.cgroup entries
PASS: No legacy 'lxc.cgroup' keys found.
INFO: Checking storage content type configuration..
PASS: no problems found
INFO: Checking if the suite for the Debian security repository is correct..
INFO: Make sure to change the suite of the Debian security repository from 'buster/updates' to 'bullseye-security' - in /etc/apt/sources.list:6
SKIP: NOTE: Expensive checks, like CT cgroupv2 compat, not performed without '--full' parameter

= SUMMARY =

TOTAL:    20
PASSED:   17
SKIPPED:  3
WARNINGS: 0
FAILURES: 0
root@prxmx024a:~#
root@prxmx024a:~#
root@prxmx024a:~#
root@prxmx024a:~# pve6to7 --full
= CHECKING VERSION INFORMATION FOR PVE PACKAGES =

Checking for package updates..
PASS: all packages uptodate

Checking proxmox-ve package version..
PASS: proxmox-ve package has version >= 6.4-1

Checking running kernel version..
PASS: expected running kernel '5.4.203-1-pve'.

= CHECKING CLUSTER HEALTH/SETTINGS =

SKIP: standalone node.

= CHECKING HYPER-CONVERGED CEPH STATUS =

SKIP: no hyper-converged ceph setup detected!

= CHECKING CONFIGURED STORAGES =

PASS: storage 'local' enabled and active.
PASS: storage 'local-lvm' enabled and active.
PASS: storage 'storageusbhdd01' enabled and active.

= MISCELLANEOUS CHECKS =

INFO: Checking common daemon services..
PASS: systemd unit 'pveproxy.service' is in state 'active'
PASS: systemd unit 'pvedaemon.service' is in state 'active'
PASS: systemd unit 'pvestatd.service' is in state 'active'
INFO: Checking for running guests..
PASS: no running guest detected.
INFO: Checking if the local node's hostname 'proxmox1' is resolvable..
INFO: Checking if resolved IP is configured on local node..
PASS: Resolved node IP '192.168.2.106' configured and active on single interface.
INFO: Checking backup retention settings..
INFO: storage 'local' - no backup retention settings defined - by default, PVE 7.x will no longer keep only the last backup, but all backups
PASS: no problems found.
INFO: checking CIFS credential location..
PASS: no CIFS credentials at outdated location found.
INFO: Checking custom roles for pool permissions..
INFO: Checking node and guest description/note legnth..
PASS: All node config descriptions fit in the new limit of 64 KiB
PASS: All guest config descriptions fit in the new limit of 8 KiB
INFO: Checking container configs for deprecated lxc.cgroup entries
PASS: No legacy 'lxc.cgroup' keys found.
INFO: Checking storage content type configuration..
PASS: no problems found
INFO: Checking if the suite for the Debian security repository is correct..
INFO: Make sure to change the suite of the Debian security repository from 'buster/updates' to 'bullseye-security' - in /etc/apt/sources.list:6
SKIP: No containers on node detected.

= SUMMARY =

TOTAL:    20
PASSED:   17
SKIPPED:  3
WARNINGS: 0
FAILURES: 0
root@prxmx024a:~#
root@prxmx024a:~# cat /etc/apt/sources.list
deb http://deb.debian.org/debian buster main contrib

deb http://deb.debian.org/debian buster-updates main contrib

# security updates
deb http://security.debian.org buster/updates main contrib

root@prxmx024a:~#
root@prxmx024a:~#
root@prxmx024a:~# sed -i 's/buster\/updates/bullseye-security/g;s/buster/bullseye/g' /etc/apt/sources.list
root@prxmx024a:~#
root@prxmx024a:~# cat /etc/apt/sources.list
deb http://deb.debian.org/debian bullseye main contrib

deb http://deb.debian.org/debian bullseye-updates main contrib

# security updates
deb http://security.debian.org bullseye-security main contrib

root@prxmx024a:~#
root@prxmx024a:~# cat /etc/apt/sources.list.d/pve-enterprise.list
deb http://download.proxmox.com/debian/pve buster pve-no-subscription
#deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise
root@prxmx024a:~#
root@prxmx024a:~# vi /etc/apt/sources.list.d/pve-enterprise.list
root@prxmx024a:~#
root@prxmx024a:~# cat /etc/apt/sources.list.d/pve-enterprise.list
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
#deb http://download.proxmox.com/debian/pve buster pve-no-subscription
#deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise
root@prxmx024a:~#
root@prxmx024a:~#
root@prxmx024a:~#
root@prxmx024a:~# apt update
Hit:1 http://deb.debian.org/debian bullseye InRelease
Get:2 http://download.proxmox.com/debian/pve bullseye InRelease [2,768 B]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Hit:4 http://security.debian.org bullseye-security InRelease
Get:5 http://download.proxmox.com/debian/pve bullseye/pve-no-subscription amd64 Packages [427 kB]
Fetched 474 kB in 0s (1,022 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
582 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@prxmx024a:~#
root@prxmx024a:~# apt list --upgradable
[...]

root@prxmx024a:~# apt dist-upgrade
[...]
root@prxmx024a:~# reboot

Hackers dont care about your...

 In 2009 there was "New Zealands Hacker con" in Wellington, which had this awesome picture:

Hackers dont care about your...

Hackers/Attackers dont care about your...

  • ...About your projects scope
  • ...It's managed by a third party
  • ...lt's a legacy system
  • ...lt's 'too critical to patcht'
  • ...About your outage windows
  • ...About your budget
  • ...You've always done it that way
  • ...About your Go-Live Date
  • ...lt's only a of concept
  • ...About Non-Disclosure Agreements
  • ...lt wasnt a requirement in the contract
  • ...lt's an intemal system
  • ...lt's really hard to change
  • ...lt's due for replacement
  • ...You're not sure how to fx it
  • ...lt's handled in the Cloud
  • ...About your Risk Register entry
  • ...The vendor doesnt support that configuration
  • ...lt's an interim solution
  • ...lt's [insert standard here]compliant
  • ...lt's encrypted on disk
  • ...The cost beneft doesn't stack up
  • ..."Nobody else could figure that Out"
  • ...You can't explain the risk to the Business
  • ...You've got Other priorities
  • ...About yourfaith in the competence of your intemal users
  • ...You dont have a business justifacation
  • ...You cant show Retum on Investment
  • ...You contracted out that risk

apt get update fails after VM snapshot reset

After you reset your VM to a snapshot including RAM you might have to time from the VM snapshot still active. So your apt get using https might fail due to the wrong time: "*update source* is not valid yet (invalid for another 46min 26s)".

Example:

[11:14:04] root@linubu22nf435:~#
[11:14:07] root@linubu22nf435:~# date
Tue 31 Oct 2023 11:14:08 PM CET
[11:14:08] root@linubu22nf435:~#
[11:14:08] root@linubu22nf435:~# apt update && apt install -y software-properties-common
Hit:1 http://ppa.launchpad.net/ondrej/php/ubuntu focal InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-security InRelease
Ign:5 https://download.webmin.com/download/repository sarge InRelease
Get:6 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Hit:7 https://download.webmin.com/download/repository sarge Release
Reading package lists... Done
E: Release file for http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease is not valid yet (invalid for another 46min 26s). Updates for this repository will not be applied.
[11:14:20] root@linubu22nf435:~#
[11:14:23] root@linubu22nf435:~# sudo hwclock --hctosys
[11:15:06] root@linubu22nf435:~#
[12:37:53] root@linubu22nf435:~# date
Tue 31 Oct 2023 12:38:27 PM CET
[12:38:27] root@linubu22nf435:~#
[12:38:28] root@linubu22nf435:~#
 

Using sudo hwclock --hctosys you update your time and apt get and apt install with https should work fine again.

Microsoft Office access does not work to WebDav shares

Since Microsoft released a patch for Windows in Q4-2023, access to WebDav shares which use basic authentication is blocked:

Example

Office Error WebDav sign-in method

Example with ionos webdav share:

English: "Microsoft Office has blocked access to https://webdav.hidrive.ionos.com because the source uses a sign-in method that may be unsecure

German: "Microsoft Office hat den Zugriff auf https://webdav.hidrive.ionos.com blockiert, da die Quelle eine Anmeldemethode verwendet, die möglicherweise unsicher ist."

 

Solution

To fix this, you have to add the following registry key to Windows 11 with the URL to your destination (in this example it is webdav.hidrive.ionos.com):

[HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Common\Identity]
"basichostallowlist"="webdav.hidrive.ionos.com" 

 

Almost perfect protection for websites and other services - Mutual TLS

Its hard to secure your IT services and applications. The list of possible attacks is long, as shown in the Mitre Att&ck framework, the OWASP lists and so on.

What helps drastically reducing the attack surface is to use mutual TLS (explanations see https://www.cloudflare.com/learning/access-management/what-is-mutual-tls/ or https://en.wikipedia.org/wiki/Mutual_authentication#mTLS or https://www.youtube.com/watch?v=x7B5CwcxCDI or https://www.ietf.org/rfc/rfc5246.txt) for authentication to establish access to your service/application. If you use certificates to authenticate your clients before accessing the first byte to of your service/application, you stop attacks at OSI layer 5. The attackers cant reach OSI layer 6 and 7 which  contain the most vulnerabilites and weaknesses by far.

I've asked ChatGPT to make a list to compare regular web applications with web applications, which use mutual TLS:

Attack TypeRegular Web ApplicationSecured Web Application (Mutual TLS)Mitre ATT&CK Tactic(s)
Cross-Site Scripting (XSS)VulnerableProtectedInitial Access, Execution, Persistence
SQL InjectionVulnerableProtectedCollection, Credential Access, Execution
Cross-Site Request Forgery (CSRF)VulnerableProtectedInitial Access, Collection
Session HijackingVulnerableProtectedCollection, Credential Access
Brute Force AttacksVulnerableProtectedCredential Access, Execution
ClickjackingVulnerableProtectedCollection, Defense Evasion
Directory TraversalVulnerableProtectedCollection, Exfiltration
File Upload VulnerabilitiesVulnerableProtectedExecution, Collection
LDAP InjectionVulnerableProtectedCollection, Credential Access
Man-in-the-Middle (MitM) AttacksVulnerableProtectedCollection, Credential Access, Execution
Remote File InclusionVulnerableProtectedExecution, Collection
XML External Entity (XXE) InjectionVulnerableProtectedCollection, Credential Access
Security MisconfigurationsVulnerableWay less vulnerableDefense Evasion, Discovery
HTTP Header InjectionVulnerableProtectedDefense Evasion, Execution
DDoS AttacksVulnerableWay less vulnerableImpact
API Security IssuesVulnerableProtectedCollection, Credential Access, Execution
HTTP Parameter PollutionVulnerableProtectedDefense Evasion, Execution
Session SidejackingVulnerableProtectedCredential Access, Collection
Social Engineering AttacksVulnerableProtectedCollection, Defense Evasion
Cross-Site Tracing (XST)VulnerableProtectedCredential Access, Collection
Fuzzing AttacksVulnerableProtectedExecution, Defense Evasion
CryptanalysisVulnerableWay less vulnerableCollection, Credential Access
Reverse EngineeringVulnerableProtectedCollection, Credential Access
Subdomain TakeoverVulnerableProtectedInitial Access, Collection
Session Token TheftVulnerableProtectedCredential Access, Collection
Vulnerabilies/0daysVulnerableWay less vulnerableInitial Access
Broken Object Level Authorization (BOLA)VulnerableProtectedCredential Access, Authorization
Improper Rate LimitingVulnerableWay less vulnerableDefense Evasion, Impact
Insecure Direct Object References (IDOR)VulnerableProtectedCollection, Credential Access
Insufficient Logging and MonitoringVulnerableWay less vulnerableDiscovery, Defense Evasion
Insecure DeserializationVulnerableProtectedExecution, Defense Evasion
Lack of Resources and Rate LimitingVulnerableProtectedDefense Evasion, Impact
Mass AssignmentVulnerableProtectedCredential Access, Collection
Insecure Cryptographic StorageVulnerableVulnerableCollection, Credential Access
Use of Components with Known VulnerabilitiesVulnerableWay less vulnerableCollection, Execution
Unvalidated Redirects and ForwardsVulnerableProtectedDefense Evasion, Initial Access
XML External Entity (XXE) InjectionVulnerableProtectedCollection, Credential Access

Caution: This is a simplified point of view - it is only focusing on the inital access. Mutual TLS certificate based authentication make the initial access for attackers almost impossible. Therefore most attacks are not possible anymore without the attackers gets his hands on a client certificate with its private key. However your IT stack is still vulnerable to OSI layer 1-5 attacks as well as e.g. to vulnerabilities within OpenSSL, LibreSSL etc.


Proxmox update error "Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1)"

Problem

During a proxmox update (e.g. from proxmox version 6 to 7) you receive the following error:

[...]
100% [608 zstd 34.1 kB/630 kB 5%] 1,337 kB/s 0s
100% [Working] 1,337 kB/s 0s

Fetched 255 MB in 2min 30s (1,702 kB/s)
W: (pve-apt-hook) !! WARNING !!
W: (pve-apt-hook) You are attempting to remove the meta-package 'proxmox-ve'!
W: (pve-apt-hook)
W: (pve-apt-hook) If you really want to permanently remove 'proxmox-ve' from your system, run the following command
W: (pve-apt-hook) touch '/please-remove-proxmox-ve'
W: (pve-apt-hook) run apt purge proxmox-ve to remove the meta-package
W: (pve-apt-hook) and repeat your apt invocation.
W: (pve-apt-hook)
W: (pve-apt-hook) If you are unsure why 'proxmox-ve' would be removed, please verify
W: (pve-apt-hook) - your APT repository settings
W: (pve-apt-hook) - that you are using 'apt full-upgrade' to upgrade your system
E: Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1)
E: Failure running script /usr/share/proxmox-ve/pve-apt-hook
root@proxmox1:~#
root@proxmox1:~# 

 

Solution

 
Proxmox VE 6.x is based on Debian 10.x which is called “buster”.
Proxmox VE 7.x is based on Debian 11.x which is called “bullseye”.  

  1. Check if your /etc/apt/sources.list.d/pve-enterprise.list file and /etc/apt/sources.list file still have "buster" (proxmox version 6) in them and replace it with "bullseye". E.g. with sed -i -e 's/buster/bullseye/g' /etc/apt/sources.list.d/pve-install-repo.list
  2. Run apt update again
  3. Run apt dist-upgrade again

Update Proxmox 6.x to latest 6.4

Update a Proxmox 6.x system to latest 6.4 using the guide https://pve.proxmox.com/wiki/Downloads#Update_a_running_Proxmox_Virtual_Environment_6.x_to_latest_6.4:

Proxmox VE 6.x is based on Debian 10.x which is called “buster”.

  1. Make sure you have a backup of all VMs, Containers, Proxmox itself etc.
  2. Login via SSH/CLI
  3. Check your sources.list file, should look like this:

    cat /etc/apt/sources.list

    deb http://deb.debian.org/debian buster main contrib
    deb http://deb.debian.org/debian buster-updates main contrib
    # security updates
    deb http://security.debian.org buster/updates main contrib

  4. Check the enterprise repository:

    cat /etc/apt/sources.list.d/pve-enterprise.list

    When running Proxmox VE 6.x with No-Subscription use:

    deb http://download.proxmox.com/debian/pve buster pve-no-subscription

    When running Proxmox VE 6.x with a subscription use:

    deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise

  5. Check Proxmox version using:

    pveversion -v


  6. Update your repository and packages:

    apt update

    If you get any errors, your sources.list (or your system or network) has a problem.
  7. Now upgrade the packages:

    apt dist-upgrade

  8. Reboot to activate the new Kernel, to check if you got all packages, run 'pveversion -v' and compare your output (all packages should have equal or higher version numbers): 
  9. Check Proxmox version using

    pveversion -v 


Example:

root@prxmx053b:~#
root@prxmx053b:~# cat /etc/apt/sources.list
deb http://deb.debian.org/debian buster main contrib
deb http://deb.debian.org/debian buster-updates main contrib
# security updates
deb http://security.debian.org buster/updates main contrib
root@prxmx053b:~#
root@prxmx053b:~#
root@prxmx053b:~# apt update
Hit:1 http://security.debian.org buster/updates InRelease
Hit:2 http://download.proxmox.com/debian/pve buster InRelease
Hit:3 http://deb.debian.org/debian buster InRelease
Hit:4 http://deb.debian.org/debian buster-updates InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
242 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@prxmx053b:~#
root@prxmx053b:~#
root@prxmx053b:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
[...]
root@prxmx053b:~#
root@prxmx053b:~# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.73-1-pve)
pve-manager: 6.4-15 (running version: 6.4-15/af7986e6)
pve-kernel-5.4: 6.4-20
pve-kernel-helper: 6.4-20
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+deb10u1
corosync: 3.1.5-pve2~bpo10+1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve2~bpo10+1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-5
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.14-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-2
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.7-pve1
root@prxmx053b:~#
root@prxmx053b:~# reboot


AKS Security - SIEM UseCase of Get Credentials

A SIEM usecase or forensic security alert for Azure Kubernetes Service AKS should be setup for az aks get-credentials as it reveals sensitive data of AKS.

Example for Azure Cloud Shell Bash:

azureksmoq [ ~ ]$
azureksmoq [ ~ ]$ az aks get-credentials --resource-group rgaks04app23 --name AKS04
Merged "AKS04" as current context in /home/azureksmoq/.kube/config
azureksmoq [ ~ ]$ 
azureksmoq [ ~ ]$
azureksmoq [ ~ ]$ cat /home/azureksmoq/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZ[...]RVJUSUZJQ0FURS0tLS0tCg==
    server: https://mykubernetescluster-dns-[...].hcp.eastus.azmk8s.io:443
  name: AKS04
contexts:
- context:
    cluster: AKS04
    user: clusterUser_rgaks04app23_AKS04
  name: AKS04
current-context: AKS04
kind: Config
preferences: {}
users:
- name: clusterUser_rgaks04app23_AKS04
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ[...]RS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJV[...]0VZLS0tLS0K
    token: bl0c8ko2[...]73m4ltf
azureksmoq [ ~ ]$ 
azureksmoq [ ~ ]$ 

This command can be helpful for developers or admins - but it is dual use as it reveals sensitive information. I recommend increasing your SIEM risk score or even make a use case with alerting.

Azure Documentation

Microsoft documentation for az aks get-credentials can be found here: https://learn.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-get-credentials or see here: https://azure.github.io/kubelogin/quick-start.html. See also https://learn.microsoft.com/en-us/azure/aks/control-kubeconfig-access

Get access credentials for a managed Kubernetes cluster.

By default, the credentials are merged into the .kube/config file so kubectl can use them. See -f parameter for details.

az aks get-credentials --name
                       --resource-group
                       [--admin]
                       [--context]
                       [--file]
                       [--format]
                       [--overwrite-existing]
                       [--public-fqdn]

Examples

Get access credentials for a managed Kubernetes cluster. (autogenerated)

az aks get-credentials --name MyManagedCluster --resource-group MyResourceGroup

Required Parameters

--name -n

Name of the managed cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--admin -a

Get cluster administrator credentials. Default: cluster user credentials.

default value: False
--context

If specified, overwrite the default context name. The --admin parameter takes precedence over --context.

--file -f

Kubernetes configuration file to update. Use "-" to print YAML to stdout instead.

default value: ~\.kube\config
--format

Specify the format of the returned credential. Available values are ["exec", "azure"]. Only take effect when requesting clusterUser credential of AAD clusters.

--overwrite-existing

Overwrite any existing cluster entry with the same name.

default value: False
--public-fqdn

Get private cluster credential with server address to be public fqdn.

default value: False
Global Parameters
--debug

Increase logging verbosity to show all debug logs.

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

--output -o

Output format.

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

 

Git pull on subfolders

In order to update your whole IDE (e.g. visual studio code), you can run git pull to all subfolders one by one using:

ls | xargs -I{} git -C {} pull

Or you run it in parallel for multiple subfolders using:

ls | xargs -P10 -I{} git -C {} pull

Example git pull one by one

DEVFLO@SYS40DEVCL001 MINGW64 /c/git
$
DEVFLO@SYS40DEVCL001 MINGW64 /c/git
$
DEVFLO@SYS40DEVCL001 MINGW64 /c/git
$ cd splunk-apps/
DEVFLO@SYS40DEVCL001 MINGW64 /c/git/splunk-apps
$ ls | xargs -I{} git -C {} pull
Already up to date.
remote: Enumerating objects: 64, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 64 (delta 15), reused 12 (delta 12), pack-reused 36
Unpacking objects: 100% (64/64), 12.38 KiB | 56.00 KiB/s, done.
From https://git.dev.domain.tld/splunk-apps/all_indexes
21ef148..dc3191d master -> origin/master
Updating 21ef148..dc3191d
Fast-forward
local/indexes.conf | 95 ++++++++++++++++++++++++++++++++++++++++++------------
1 file changed, 75 insertions(+), 20 deletions(-)
remote: Enumerating objects: 19, done.
remote: Counting objects: 100% (19/19), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 16 (delta 8), reused 6 (delta 2), pack-reused 0
Unpacking objects: 100% (16/16), 2.00 KiB | 34.00 KiB/s, done.
From https://git.dev.domain.tld/splunk-apps/hf_inputs_http
45221cc..9ba7459 master -> origin/master
Updating 45221cc..9ba7459
Fast-forward
local/inputs.conf | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 72 insertions(+), 2 deletions(-)
remote: Enumerating objects: 314, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 314 (delta 17), reused 5 (delta 5), pack-reused 281
[...]

Example git pull in parallel

DEVFLO@SYS40DEVCL001 MINGW64 /c/git/splunk-apps
$
DEVFLO@SYS40DEVCL001 MINGW64 /c/git/splunk-apps
$
DEVFLO@SYS40DEVCL001 MINGW64 /c/git/splunk-apps
$ ls | xargs -P10 -I{} git -C {} pull
Already up to date.
Already up to date.
Already up to date.
Already up to date.
Already up to date.
Already up to date.
remote: Enumerating objects: 101, done.
remote: Counting objects: 100% (76/76), done.
remote: Compressing objects: 100% (38/38), done.
remote: Total 101 (delta 46), reused 53 (delta 35), pack-reused 25
Receiving objects: 100% (101/101), 101.59 KiB | 12.70 MiB/s, done.
Resolving deltas: 100% (61/61), completed with 1 local object.
From https://git.dev.domain.tld/splunk-apps/search_securitywork
7c8b27a..913ed1e master -> origin/master
Updating 7c8b27a..913ed1e
Fast-forward
{local => default}/data/ui/nav/default.xml | 0
.../ui/views/security__asset_information.xml | 4 +-
.../ui/views/security__dashkpi1__logsource.xml | 4 +-
.../ui/views/security__dashkpi2__usecases.xml | 4 +-
.../views/security__dashkpi3__technology.xml | 2 +-
.../ui/views/security__dashkpi4_logvolume.xml | 2 +-
{local => default}/macros.conf | 0
{local => default}/mlspl.conf | 0
{local => default}/savedsearches.conf | 160 +++++++++++++++++----
{local => default}/transforms.conf | 0
lookups/uc_info.csv | 8 +-
14 files changed, 152 insertions(+), 50 deletions(-)
rename {local => default}/data/ui/nav/default.xml (100%)
rename {local => default}/data/ui/views/security__asset_information.xml (94%)
rename {local => default}/data/ui/views/security__dashkpi1__logsource.xml (99%)
rename {local => default}/data/ui/views/security__dashkpi2__usecases.xml (99%)
rename {local => default}/data/ui/views/security__dashkpi3__technology.xml (97%)
rename {local => default}/data/ui/views/security__dashkpi4_logvolume.xml (99%)
[...]

Microsoft Portals overview - msportals.io

The website msportals.io is listing a nice overview of Microsofts portals. For example administrator portals:

Microsoft 365 Admin Portals


Many more can be found on
msportals.io.

 

Mitre D3fend Framework - Att&ck from a defenders perspective

Mitre published another awesome framework called d3fend.mitre.org

Mitre D3fend

It is using the att&ck framework but from a defenders perspective :-) 

D3fend Att&ck relationship



Confluence behind LoadBalancer with another domain results in XSRF error

If you have an atlassian confluence running, which is published by a loadbalancer or reverse proxy using another domain, you might run into an XSRF error.

Example

Confluence FQDN: somehostname.domain.tld
LoadBalancer Confluence FQDN: confluence.domain.tld

Some actions like uploading your profile picture (https://confluence.domain.tld/users/profile/editmyprofilepicture.action) do not work. You'll receive an generic error from the confluence page (see red box of the screenshot below). If you check the HTTP Header response, you'll see XSRF check failed. It is caused by the confluence cross site request forgery (CSRF) protection.

Confluence XSRF Error

Solution

Edit confluence server.xml and add the FQDN from the LoadBalancer or reverse proxy.

More information can be found here: https://confluence.atlassian.com/kb/cross-site-request-forgery-csrf-protection-changes-in-atlassian-rest-779294918.html



Splunk Enterprise update plan

Splunk published this awesome Splunk Enterprise update plan: https://docs.splunk.com/images/d/d3/Splunk_upgrade_order_of_ops.pdf 

Regardless if you have a single-site or multi-site splunk installation, if your are running a stand-alone or distributed and/or clustered architecture, if you are using Splunks Universal Forwarder, the Deployment server, a License Master, Search Head cluster or Indexer Cluster master or not - this plan has your environment setup covered.

Step by step it guides you in updating your Splunk Enterprise environment including backuping up every system, checking each systems health and possible connectivity issues as well as the updates itself, may it be a simple upgrade or a rolling upgrade. Additional informations about each step can be found in the PDF as a link to docs.splunk.com.

Splunk Enterprise update plan step by step


 

Phishing using @-URL trick in DNS .zip domains

With Googles release of the DNS top-level-domains .zip and .mov a new phishing (mitre att&ck T1566) trick is possible as bobbyrsec wrote about.

Example 1

https://www.google.com/?q=example.text  <— FQDN = google.com

Example 2

https://www.google.com/example/text/@v1271.zip <— FQDN = google.com right? No, it is v1271.zip. Because the @ character describes e.g. the authentication of the URL.

Example 3

https://www.google.com/example/text/v1271.zip <— FQDN = google.com

Reason



Result

So doublechecking URLs becomes harder. Using Fido2, Passkeys or password-managers (e.g. bitwarden.com) with auto-fill becomes more important because they dont fall for that trick and are more phishing-resistant.

Overview of public interfaces for SOC/IT-Security staff

In case of an IT-security incident, emergency oder if a new critical vulnerability (like log4j in December 2021) arises, it is good to be prepared, so you can quickly answer questions like:

  • "Are we affected?"
  • "Do we use this technology?"
  • "Where do we use this vulnerable protocol?"
  • "To whom is the attack surface exposed to?"
  • "Are there mitigations in place?"
  • "Is is exploitable without authentication in our setup?"
  • "Which is the best place to place a first mitigation?"
  • etc..
An overview like the following can and will be helpful for your IT-security staff or your Security Operations Center SOC:

System Internet Facing Protocol Authentication Security Used Products/Vendors Logs send to SIEM Contact Person Known Weaknesses
Websites Yes, exposed to all public-ip-addresses HTTPS (TCP:443) & HTTP (TCP:80 - HTTP 301 Redirect to HTTPS) None Web Application Firewall F5 BigIP LoadBalancer WAF & Apache Container on OpenShift Yes Link to CMDB Websites may contain 3rd party code, SBOM see CMDB
Managed File Transfer Yes, but limited to dedicated public ip-addresses of partners HTTPS (TCP:443) HTTPS Tokens Web Application Firewall F5 BigIP LoadBalancer WAF IPSwitch Yes Link to CMDB Runs on VM as appliance, OS might not be hardend from vendor
Citrix Yes, exposed to all public-ip-addresses HTTPS (TCP:443) MFA Netscaler WAF Citrix Systems + Okta MFA Yes Link to CMDB NetScaler WAF Ruleset might be out-of-date
Mailserver Yes, exposed to all public-ip-addresses SMTP (TCP:25) None AntiSpam Mailgatway & AV-Sandbox Cisco E-Mail Security Yes Link to CMDB Mailgateways run on Hardware, might not be hardended from vendor
SSLVPN S2E Yes, exposed to all public-ip-addresses HTTPS (TCP:443) Mutual TLS Certbased + MFA Azure DDoS FortiGate SSLVPN Azure VM + Okta MFA Yes Link to CMDB Possible FortiGate FortiOS SSLVPN Vulnerabilities
M365 ActiveSync Yes, exposed to all public-ip-addresses HTTPS (TCP:443) Mutual TLS Certbased Azure DDoS Microsoft 365 + Intunes Yes Link to CMDB Not part of own Vulnerability-Scanner
VPN S2S Yes, but limited to dedicated public ip-addresses of partners IPSec UDP:500 & UDP:4500 & ESP IPsec IKEv2 Certbased Auth Azure DDoS FortiGate SSLVPN Azure VM Link to CMDB -
DMARC SaaS Yes, exposed to all public-ip-addresses DNS (UDP:53), HTTP (TCP:80), HTTPS (TCP:443), SMTP (TCP:25) None - dmarcadvisor.com SaaS No Link to CMDB Not part of own Vulnerability-Scanner
DNS Server Yes, but limited to dedicated public ip-addresses of partners DNS (UDP:53 & TCP:53) None Azure Network Security Groups RHEL Bind Yes Link to CMDB -
ISP Routers Yes, but limited to dedicated public ip-addresses of ISP routers BGP (TCP:179), BFD, Ping (ICMP:0/8) BGP MD5 Auth - Extreme Networks XOS Yes Link to CMDB
etc.. etc.. etc.. etc.. etc.. etc.. etc.. etc.. etc..

 

Of course you can add many more columns like e.g.:

  • "SBOM technologys used" (for example: RHEL, Apache Tomcat, OpenSSL, log4j, puppet, ansible, splunk universal forwarder, appdynamics,..)
  • Direct links to your Firewall Management System, WAF or SIEM
  • "Is it part of our vulnerability scanner?"
  • "Is the vulnerability scanner scanning it authenticated?"
  • "Is the system/application hardended?"
  • and so on :-)
This list will help in case of an IT-security emergency to sort out the first steps in order to mitigate and fix the issue of the public exposed interfaces (like to the internet or to business partners). However this is only one of many steps necessary - always "asume breach" and make sure an attacker controlling a client or server still is unable to spread (unnoticed) in your companies (cloud) network.

Windows fileless persistence local privilege escalation using sc.exe sdset

Alh4zr3d and Grzegorz Tworek wrote about a fileless, local privilege escalation backdoor in order to gain persistence (TA0003) using the following Windows command:

sc.exe sdset scmanager D:(A;;KA;;;WD) 

Windows sc.exe persistence fileless backdoor

Setting the security descriptor on the service manager allows anyone to start SYSTEM services.

To show the security descriptor of service manager:

sc.exe sdshow scmanager

 

The Mitre Att&ck framework has technique T1569.002 for this "System Services: Service Execution", writing: "Adversaries may abuse the Windows service control manager to execute malicious commands or payloads. The Windows service control manager (services.exe) is an interface to manage and manipulate services. The service control manager is accessible to users via GUI components as well as system utilities such as sc.exe and Net."

Microsofts own documentation about sc.exe sdset:

Sets a service's security descriptor, using the Service Descriptor Definition Language (SDDL).

Syntax

sc [<ServerName>] sdset <ServiceName> <ServiceSecurityDescriptor>

Parameters

Parameter

Description

<ServerName>

Specifies the name of the remote server on which the service is located. The name must use the Universal Naming Convention (UNC) format (for example, \\myserver). To run SC.exe locally, omit this parameter.

<ServiceName>

Specifies the service name returned by the getkeyname operation.

<ServiceSecurityDescriptor>

Specifies the service descriptor in SDDL.

/?

Displays help at the command prompt.


To explain the DACL D:(A;;KA;;;WD) gav_gall asked ChatGPT to explain:

ChatGPT DACL Windows persistence backdoor


Filter logs in Splunk - example filtering monitor probe checks

When running Splunk you want to filter logs, for example to get rid of the many health check probe querys from your monitoring system. Examp...