The following shows a splunk savedsearch/alert which searches for Splunk buckets which are not correctly replicated to all indexers.
Example
For example if you have a multisite cluster having 2 sites and each site should contain 2 copies of a bucket:
splunk_server_clustering_available_sites: "site1,site2"
splunk_server_clustering_site_replication_factor: 'origin:1, site1:2, site2:2, total:4'
splunk_server_clustering_site_search_factor: 'origin:1, site1:2, site2:2, total:4'
Then the following SPL or savedsearch/alert might help identify if multiple buckets of an index are only replicated once:
| dbinspect index=* ```<-- show all buckets of all indexes ```
|search NOT state=hot ```<-- only warm & cold buckets ```
|eventstats count by bucketId ```<-- list all bucket-ids only once, count how often they occur ```
|search count<2 ```<-- filter for all buckets that occur only once and are not replicated 4 times ```
|stats count by index ```<-- show all indexes that have buckets which were replicated only once ```
|search count>10 ```<-- show all indexes that have more than 10 buckets which were replicated only once```
``` All buckets should be replicated 4 times according to the search/replication factor of the Splunk multisite cluster. This alert shows if there are indexes with over 10 buckets that are only present once instead of being replicated on 4 indexers```
Screenshot:
Explaining screenshot:
