Splunk alert for buckets which are not correctly replicated

The following shows a splunk savedsearch/alert which searches for Splunk buckets which are not correctly replicated to all indexers. 

Example

For example if you have a multisite cluster having 2 sites and each site should contain 2 copies of a bucket: 

splunk_server_clustering_available_sites: "site1,site2"
splunk_server_clustering_site_replication_factor: 'origin:1, site1:2, site2:2, total:4'
splunk_server_clustering_site_search_factor: 'origin:1, site1:2, site2:2, total:4'


Then the following SPL or savedsearch/alert might help identify if multiple buckets of an index are only replicated once:

| dbinspect index=* ```<-- show all buckets of all indexes ``` 
|search NOT state=hot ```<-- only warm & cold buckets ``` 
|eventstats count by bucketId  ```<-- list all bucket-ids only once, count how often they occur ``` 
|search count<2 ```<-- filter for all buckets that occur only once and are not replicated 4 times ``` 
|stats count by index ```<-- show all indexes that have buckets which were replicated only once ``` 
|search count>10 ```<-- show all indexes that have more than 10 buckets which were replicated only once```
``` All buckets should be replicated 4 times according to the search/replication factor of the Splunk multisite cluster. This alert shows if there are indexes with over 10 buckets that are only present once instead of being replicated on 4 indexers``` 


Screenshot:

Splunk bucket only once replicated dbinspect

Explaining screenshot:

Splunk bucket only once replicated dbinspect


No comments:

Post a Comment

Splunk UseCase for attacks against FortiGate Firewall management interfaces

If you are using Splunk as your SIEM you can try to detect attacks against your FortiGate firewalls by using the following SPL query: index...