Filter logs in Splunk - example filtering monitor probe checks

When running Splunk you want to filter logs, for example to get rid of the many health check probe querys from your monitoring system.

Example filtering PRTG monitoring probe requests using props.conf and transforms.conf

1. Find the monitoring probes in the logs in splunk, e.g.:

10.148.227.111 - - [18/Jul/2024:23:21:06 +0200] "GET /login HTTP/1.1" 200 12882 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:21:06 +0200] "GET / HTTP/1.1" 302 5793 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:20:56 +0200] "GET /login HTTP/1.1" 200 12882 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:20:56 +0200] "GET / HTTP/1.1" 302 5790 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.121 - - [18/Jul/2024:23:12:17 +0200] "GET /login HTTP/1.1" 200 17480 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com; 576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:16 +0200] "GET / HTTP/1.1" 302 5572 "-" Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com; 576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:15 +0200] "GET /login HTTP/1.1" 200 17486 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com; 576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:15 +0200] "GET /login HTTP/1.1" 200 17474 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com; 576bb8887fe66b1eece876e62e701b9e)"

2. Create a regex, which finds the logs (which a precise match but as less cpu steps as possible) using https://regex101.com/

In this example the following regexes where used:

Mozilla\/\d+\.\d+\s+\(compatible;\s+PRTG\s+Network\s+Monitor
Mozilla\/\d.\d\s\(compatible\;\sPaesslerCloudBot\/\d.\d
 

3. Create a dedicated splunk app for this log source or use the default splunk search app and modify the props.conf. Create an entry which you map to the host, source or sourcetype and tell it to use transforms.conf:

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#
uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local# cat props.conf
[...]

#filter prtg monitoring logs
[host::fqdn.of.logsource]
TRANSFORMS-t1=filter-prtg-from-access
TRANSFORMS-t2=filter-prtgcloud-from-access

4. Modify the transforms.conf of this same splunk app. Create an entry which you map to the host, source or sourcetype and force it to the nullQueue:

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#
uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local# cat transforms.conf
#filter prtg logs von access.log von nextcloud
#
[filter-prtg-from-access]
REGEX = Mozilla\/\d.\d\s\(compatible\;\sPRTG\sNetwork\sMonitor
DEST_KEY = queue
FORMAT = nullQueue

[filter-prtgcloud-from-access]
REGEX = Mozilla\/\d.\d\s\(compatible\;\sPaesslerCloudBot\/\d.\d
DEST_KEY = queue
FORMAT = nullQueue

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#

5. Reload the splunk configuration using https://your.splunk.fqdn:8000/en-GB/debug/refresh 

6. Your logs should be filtered. If not, check the btool to see if another splunk configuration takes precedence to your configuration:

./splunk btool props list
./splunk btool props list --debug
./splunk btool transforms list
./splunk btool transforms list --debug