Showing posts with label Splunk. Show all posts
Showing posts with label Splunk. Show all posts

Splunk GUI enhancement - 100% client-side - browser DOM manipulation

Problem

Cisco Splunk Enterprise is awesome and the Splunk GUI is, too. However sometimes you need to quickly search the result data.

Solution

My coworker and friend Malte created a 100% client-side browser bookmarklet (see https://de.wikipedia.org/wiki/Bookmarklet) which enables you to search your data in the browser GUI fast and easily. Works fine with Firefox and Chrome with Splunk Enterprise v10.0 and v10.2.
 
Demo:
Splunk GUI enhancement GIF

How it works under the hood

There is no magic - just client-side browser DOM work:
  1. It finds Splunks classic result tables by their CSS classes (table-chrome, results-table, and friends).
  2. It injects a scoped stylesheet (all classes are namespaced with bm-, for bookmarklet) so it never clobbers Splunk's own styles.
  3. Filtering is pure client-side: it reads each cell's text, compares it against your tags, and toggles row visibility.
  4. Page merging watches the paginator, waits for the active page to change, then splices the new rows back into the master table.

Everything is client-side: no external network calls, no telemetry, no data leaves the page.
 

Caveats

It is built for Cisco Splunks classic UI. So it won't work on the newer Splunk Dashboard Studio dashboards. And the page-merge feature does re-trigger searches as it clicks through — handy, but worth using gently on a busy shared search head. 
 

Installation 

  1. Open Browser (Chrome, Firefox)
  2. Create a bookmark and insert the following javascript as URL:



    https://github.com/flostyen/SplunkMalteBeautifier/blob/main/bookmarklet


    javascript:(function(){const splunkTables=document.querySelectorAll('.results-table .table, table.table-chrome');if(splunkTables.length===0){alert('No classic Splunk dashboard or search tables found.');return}if(!document.getElementById('splunk-bookmarklet-styles')){const style=document.createElement('style');style.id='splunk-bookmarklet-styles';style.innerHTML='.bm-top-bar { display: flex; align-items: center; margin-bottom: 5px; flex-wrap: wrap; gap: 8px; } .bm-stats-label { margin-left: 5px; font-size: 12px; color: #555; } .bm-filter-row th { border-bottom: 1px solid #ccc !important; background: #f5f5f5; padding: 4px !important; vertical-align: top; font-weight: normal; } .bm-filter-input { width: 100%; border: 1px solid #ccc; border-radius: 3px; padding: 2px 4px; font-size: 11px; box-sizing: border-box; } .bm-filter-input:focus { outline: none; border-color: #5cc05c; } .bm-settings-btn { cursor: pointer; background: #eee; border: 1px solid #ccc; padding: 0 8px; border-radius: 4px; font-size: 12px; height: 26px; display: inline-flex; align-items: center; box-sizing: border-box; } .bm-settings-btn:hover { background: #ddd; } .bm-merge-group { display: flex; align-items: stretch; border: 1px solid #ccc; border-radius: 4px; overflow: hidden; height: 26px; box-sizing: border-box; } .bm-merge-label { display: flex; align-items: center; padding: 0 6px; font-size: 12px; background: #f5f5f5; color: #555; border-right: 1px solid #ccc; } .bm-merge-input { width: 40px; border: none !important; padding: 0; font-size: 13px; font-weight: bold; text-align: center; outline: none; box-sizing: border-box; box-shadow: none !important; } .bm-merge-input::-webkit-outer-spin-button, .bm-merge-input::-webkit-inner-spin-button { -webkit-appearance: none; margin: 0; } .bm-merge-input[type=number] { -moz-appearance: textfield; } .bm-merge-btn { cursor: pointer; background: #dbeafe; color: #1e40af; border: none; border-left: 1px solid #a5b4fc; padding: 0 8px; font-size: 12px; font-weight: bold; } .bm-merge-btn:hover { background: #bfdbfe; } .bm-merge-btn:disabled, .bm-merge-input:disabled { opacity: 0.6; cursor: not-allowed; background: #f3f4f6; } .bm-tag-container { display: flex; flex-wrap: wrap; gap: 2px; margin-top: 4px; } .bm-tag { display: inline-flex; align-items: center; background: #e0e7ff; border: 1px solid #a5b4fc; border-radius: 3px; padding: 1px 4px; font-size: 10px; color: #3730a3; word-break: break-all; } .bm-tag.bm-tag-negative { background: #fee2e2; border-color: #fca5a5; color: #991b1b; } .bm-tag-remove { cursor: pointer; font-weight: bold; margin-left: 4px; opacity: 0.6; } .bm-tag-remove:hover { opacity: 1; } .bm-modal-overlay { position: fixed; top: 0; left: 0; width: 100%; height: 100%; background: rgba(0,0,0,0.5); z-index: 999999; display: flex; align-items: center; justify-content: center; } .bm-modal { background: #fff; padding: 15px; border-radius: 6px; min-width: 300px; max-height: 80vh; overflow-y: auto; font-size: 13px; box-shadow: 0 4px 12px rgba(0,0,0,0.2); } .bm-modal h3 { margin-top: 0; margin-bottom: 10px; font-size: 16px; border-bottom: 1px solid #eee; padding-bottom: 5px; } .bm-modal label { display: block; margin-bottom: 5px; cursor: pointer; } .bm-modal-close { margin-top: 15px; background: #0073e7; color: white; border: none; padding: 5px 10px; border-radius: 4px; cursor: pointer; width: 100%; } .bm-modal-close:hover { background: #005bb5; } mark.bm-highlight { background-color: #ffeb3b !important; color: #000 !important; border-radius: 2px; padding: 0 2px; }';document.head.appendChild(style)}function toggleColumn(table,colIndex,isVisible){const displayStyle=isVisible?'':'none';const theadTrs=table.querySelectorAll('thead tr');theadTrs.forEach(tr=>{const th=tr.querySelector(`th:nth-child(${colIndex+1})`);if(th)th.style.display=displayStyle});const tbodyTrs=table.querySelectorAll('tbody tr');tbodyTrs.forEach(tr=>{const td=tr.querySelector(`td:nth-child(${colIndex+1})`);if(td)td.style.display=displayStyle})}function createTag(container,term,table,filterRow){const isNegative=term.startsWith('!');const tag=document.createElement('span');tag.className='bm-tag'+(isNegative?' bm-tag-negative':'');tag.dataset.term=term;const text=document.createElement('span');text.textContent=term;tag.appendChild(text);const removeBtn=document.createElement('span');removeBtn.className='bm-tag-remove';removeBtn.innerHTML='&#10005;';removeBtn.onclick=()=>{tag.remove();applyFilters(table,filterRow)};tag.appendChild(removeBtn);container.appendChild(tag)}function createInputNodes(table,filterRow){const input=document.createElement('input');input.type='text';input.className='bm-filter-input';input.placeholder='Enter tags...';input.title='Press Enter to create tags (! for exclude)';const tagContainer=document.createElement('div');tagContainer.className='bm-tag-container';input.onkeydown=(e)=>{if(e.key==='Enter'){e.preventDefault();const val=input.value.trim();if(val){const terms=val.split(/\s+/).filter(t=>t.length>0);terms.forEach(term=>createTag(tagContainer,term,table,filterRow));input.value='';applyFilters(table,filterRow)}}};input.onkeyup=(e)=>{if(e.key!=='Enter'){applyFilters(table,filterRow)}};return{input,tagContainer}}function removeHighlights(root){const marks=Array.from(root.querySelectorAll('mark.bm-highlight'));marks.forEach(mark=>{const parent=mark.parentNode;if(parent){parent.replaceChild(document.createTextNode(mark.textContent),mark);parent.normalize()}})}function highlightNode(node,terms){if(node.nodeType===3){const text=node.nodeValue;const lowerText=text.toLowerCase();let matchFound=false;let matchStart=-1;let matchLength=0;for(let i=0;i<terms.length;i++){const term=terms[i];if(!term)continue;const idx=lowerText.indexOf(term);if(idx!==-1){if(!matchFound||idx<matchStart){matchStart=idx;matchLength=term.length;matchFound=true}}}if(matchFound){const matchText=text.substring(matchStart,matchStart+matchLength);const beforeText=text.substring(0,matchStart);const afterText=text.substring(matchStart+matchLength);const mark=document.createElement('mark');mark.className='bm-highlight';mark.textContent=matchText;const parent=node.parentNode;if(!parent||parent.tagName==='SCRIPT'||parent.tagName==='STYLE'||parent.tagName==='MARK')return;if(beforeText){parent.insertBefore(document.createTextNode(beforeText),node)}parent.insertBefore(mark,node);const afterNode=document.createTextNode(afterText);parent.insertBefore(afterNode,node);parent.removeChild(node);highlightNode(afterNode,terms)}}else if(node.nodeType===1&&node.childNodes&&!/(script|style|mark)/i.test(node.tagName)){Array.from(node.childNodes).forEach(child=>highlightNode(child,terms))}}function enhanceTable(table,tableIndex){table.dataset.bmIndex=tableIndex;table.bmAccumulatedRows=table.bmAccumulatedRows||[];const isEventTable=table.classList.contains('events-results-table')||table.classList.contains('events-table');const thead=table.querySelector('thead');if(!thead)return;const headerRow=thead.querySelector('tr');if(!headerRow)return;if(headerRow.nextElementSibling&&headerRow.nextElementSibling.classList.contains('bm-filter-row'))return;const thElements=Array.from(headerRow.querySelectorAll('th'));const colNames=thElements.map(th=>{const firstA=th.querySelector('a');return firstA?firstA.textContent.trim():th.textContent.trim()});const topBar=document.createElement('div');topBar.className='bm-top-bar';const settingsBtn=document.createElement('button');settingsBtn.innerHTML='\u2699\uFE0F Column Settings';settingsBtn.className='bm-settings-btn';settingsBtn.onclick=(e)=>{e.preventDefault();openSettings(table,colNames)};const mergeGroup=document.createElement('div');mergeGroup.className='bm-merge-group';const mergeLabel=document.createElement('span');mergeLabel.className='bm-merge-label';mergeLabel.innerHTML='\u2795 Next';const mergeInput=document.createElement('input');mergeInput.type='number';mergeInput.min='1';mergeInput.value='1';mergeInput.className='bm-merge-input';mergeInput.title='Number of pages to merge';const mergeBtn=document.createElement('button');mergeBtn.innerHTML='Pages';mergeBtn.className='bm-merge-btn';mergeGroup.appendChild(mergeLabel);mergeGroup.appendChild(mergeInput);mergeGroup.appendChild(mergeBtn);const statsLabel=document.createElement('span');statsLabel.id=`bm-stats-${tableIndex}`;statsLabel.className='bm-stats-label';const filterRow=document.createElement('tr');filterRow.className='bm-filter-row';mergeBtn.onclick=(e)=>{e.preventDefault();const pagesToMerge=parseInt(mergeInput.value,10)||1;if(pagesToMerge<1)return;mergeBtn.disabled=true;mergeInput.disabled=true;function doMerge(remaining){if(remaining<=0){mergeBtn.innerHTML='Pages';mergeBtn.disabled=false;mergeInput.disabled=false;return}mergeBtn.innerHTML=`\u23F3 (${remaining})...`;let paginator=null;let pNode=table.parentElement;while(pNode&&pNode!==document.body){paginator=pNode.querySelector('.splunk-paginator, .shared-searchresultspaginator');if(paginator)break;pNode=pNode.parentElement}if(!paginator){alert('Paginator not found for this table.');mergeBtn.innerHTML='Pages';mergeBtn.disabled=false;mergeInput.disabled=false;return}const nextBtn=paginator.querySelector('a[data-page="next"], li.next a');let isDisabled=!nextBtn;if(nextBtn){if(nextBtn.classList.contains('disabled')||nextBtn.getAttribute('aria-disabled')==='true'||(nextBtn.parentElement&&nextBtn.parentElement.classList.contains('disabled'))){isDisabled=true}}if(isDisabled){mergeBtn.innerHTML='Pages';mergeBtn.disabled=false;mergeInput.disabled=false;return}const currentActive=paginator.querySelector('a.selected, li.active a');const currentActivePage=currentActive?currentActive.textContent.trim():null;const tbody=table.querySelector('tbody');if(tbody){const nativeRows=Array.from(tbody.querySelectorAll('tr:not(.bm-filter-row):not(.bm-merged-row)'));nativeRows.forEach(r=>{const clone=r.cloneNode(true);clone.classList.add('bm-merged-row');table.bmAccumulatedRows.push(clone)})}nextBtn.click();let attempts=0;const checkInterval=setInterval(()=>{attempts++;const newActive=paginator.querySelector('a.selected, li.active a');const newActivePage=newActive?newActive.textContent.trim():null;if(newActivePage!==currentActivePage){clearInterval(checkInterval);setTimeout(()=>{const newTbody=table.querySelector('tbody');if(newTbody){table.bmAccumulatedRows.slice().reverse().forEach(tr=>{newTbody.insertBefore(tr,newTbody.firstChild)})}if(!table.querySelector('.bm-filter-row')){const currentThead=table.querySelector('thead');if(currentThead)currentThead.appendChild(filterRow)}if(!isEventTable){const ths=table.querySelectorAll('thead tr:first-child th');ths.forEach((th,idx)=>{if(th.style.display==='none')toggleColumn(table,idx,false)})}applyFilters(table,filterRow);doMerge(remaining-1)},300)}else if(attempts>50){clearInterval(checkInterval);alert('Timed out waiting for Splunk to load the next page.');mergeBtn.innerHTML='Pages';mergeBtn.disabled=false;mergeInput.disabled=false}},200)}doMerge(pagesToMerge)};if(!isEventTable){topBar.appendChild(settingsBtn)}topBar.appendChild(mergeGroup);topBar.appendChild(statsLabel);if(table.parentElement.classList.contains('shared-resultstable-resultstablemaster')||table.parentElement.classList.contains('table-wrapper')){table.parentElement.parentNode.insertBefore(topBar,table.parentElement)}else{table.parentNode.insertBefore(topBar,table)}thElements.forEach((th)=>{const filterCell=document.createElement('th');filterCell.style.cssText=th.style.cssText;filterCell.className=th.className;filterCell.classList.remove('sorts');filterCell.style.padding='4px';const colText=th.textContent.trim();if(isEventTable){if(colText==='Event'||colText==='Time'){const nodes=createInputNodes(table,filterRow);filterCell.appendChild(nodes.input);filterCell.appendChild(nodes.tagContainer)}}else{const nodes=createInputNodes(table,filterRow);filterCell.appendChild(nodes.input);filterCell.appendChild(nodes.tagContainer)}filterRow.appendChild(filterCell)});thead.appendChild(filterRow);applyFilters(table,filterRow)}function applyFilters(table,filterRow){const isEventTable=table.classList.contains('events-results-table')||table.classList.contains('events-table');const tbody=table.querySelector('tbody');if(!tbody)return;const trs=tbody.querySelectorAll('tr:not(.bm-filter-row)');const filterCells=filterRow.querySelectorAll('th');let visibleCount=0;const totalCount=trs.length;trs.forEach(tr=>{let rowVisible=true;removeHighlights(tr);const cellHighlightMap=new Map();const tds=tr.querySelectorAll('td');filterCells.forEach((cell,index)=>{if(!rowVisible)return;const input=cell.querySelector('input');if(!input)return;const tagContainer=cell.querySelector('.bm-tag-container');let terms=[];const tags=tagContainer.querySelectorAll('.bm-tag');tags.forEach(tag=>terms.push(tag.dataset.term));const inputText=input.value.trim().toLowerCase();if(inputText){terms=terms.concat(inputText.split(/\s+/).filter(t=>t.length>0))}if(terms.length>0){let cellText='';if(isEventTable){cellText=tr.textContent.toLowerCase()}else{const td=tds[index];if(td)cellText=(td.textContent||td.innerText).toLowerCase()}if(cellText){for(let i=0;i<terms.length;i++){const term=terms[i].toLowerCase();const isNegative=term.startsWith('!');const actualTerm=isNegative?term.substring(1):term;if(actualTerm==='')continue;const hasTerm=cellText.includes(actualTerm);if(isNegative&&hasTerm){rowVisible=false;break}if(!isNegative&&!hasTerm){rowVisible=false;break}}if(rowVisible){const posTerms=terms.filter(t=>!t.startsWith('!')).map(t=>t.toLowerCase());if(posTerms.length>0){if(isEventTable){if(!cellHighlightMap.has(tr))cellHighlightMap.set(tr,new Set());posTerms.forEach(t=>cellHighlightMap.get(tr).add(t))}else{const td=tds[index];if(td){if(!cellHighlightMap.has(td))cellHighlightMap.set(td,new Set());posTerms.forEach(t=>cellHighlightMap.get(td).add(t))}}}}}else{rowVisible=false}}});if(rowVisible){visibleCount++;cellHighlightMap.forEach((termSet,element)=>{highlightNode(element,Array.from(termSet))})}tr.style.display=rowVisible?'':'none'});const statsLabel=document.getElementById(`bm-stats-${table.dataset.bmIndex}`);if(statsLabel){const hiddenCount=totalCount-visibleCount;statsLabel.innerHTML=`Showing <strong>${visibleCount}</strong> of <strong>${totalCount}</strong> rows (<strong>${hiddenCount}</strong> filtered)`}}function openSettings(table,colNames){const overlay=document.createElement('div');overlay.className='bm-modal-overlay';const modal=document.createElement('div');modal.className='bm-modal';modal.innerHTML='<h3>\u2699\uFE0F Toggle Columns</h3>';colNames.forEach((name,idx)=>{const label=document.createElement('label');const cb=document.createElement('input');cb.type='checkbox';const th=table.querySelector(`thead tr th:nth-child(${idx+1})`);cb.checked=th&&th.style.display!=='none';cb.onchange=(e)=>{toggleColumn(table,idx,e.target.checked)};label.appendChild(cb);label.appendChild(document.createTextNode(' '+(name||`Column ${idx+1}`)));modal.appendChild(label)});const closeBtn=document.createElement('button');closeBtn.className='bm-modal-close';closeBtn.innerHTML='Close';closeBtn.onclick=()=>overlay.remove();modal.appendChild(closeBtn);overlay.appendChild(modal);document.body.appendChild(overlay)}splunkTables.forEach((table,index)=>enhanceTable(table,index))})();

    Or see here: https://raw.githubusercontent.com/flostyen/SplunkMalteBeautifier/refs/heads/main/bookmarklet

  3. Open Splunk GUI and execute a search
  4. Click on the bookmark
  5. The bookmark javascript code is executed on the Splunk GUI local cache of your browser 

 

Disabled Transparent Huge Pages with created systemd service

In some applications (e.g. Splunk) you need to disable Transparent Huge Pages THP

When your systems are running in a managed environment (e.g. the public cloud - like Microsoft Azure) with cloud images (like "0001-com-ubuntu-server-jammy" or "0001-com-ubuntu-confidential-vm-jammy" (both Ubuntu22.04 LTS)), you may not be able to use the GRUB Edit to disable THP, because the cloud image ignores the GRUB edits. A possible solution can be a custom systemd service:

Example Disable Transparent Huge Pages once (not reboot-persistentšŸ’¢):

user@devazubu227:~$
user@devazubu227:~$
user@devazubu227:~$ sudo cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
user@devazubu227:~$
user@devazubu227:~$
user@devazubu227:~$ echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
never
user@devazubu227:~$
user@devazubu227:~$ sudo cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
user@devazubu227:~$



Example Disable Transparent Huge Pages with custom systemd service (reboot-persistent ✅):


Commands:

sudo tee /etc/systemd/system/disable-thp.service > /dev/null <<EOF
[Unit]
Description=Disable Transparent Huge Pages
After=network.target

[Service]
Type=simple
ExecStart=/bin/bash -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl enable disable-thp
sudo systemctl start disable-thp
sudo systemctl status disable-thp
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag



Example

user@devazubu227:~$
user@devazubu227:~$
user@devazubu227:~$ sudo tee /etc/systemd/system/disable-thp.service > /dev/null <<EOF
[Unit]
Description=Disable Transparent Huge Pages
After=network.target

[Service]
Type=simple
ExecStart=/bin/bash -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target
EOF
user@devazubu227:~$
user@devazubu227:~$ cat /etc/systemd/system/disable-thp.service
[Unit]
Description=Disable Transparent Huge Pages
After=network.target

[Service]
Type=simple
ExecStart=/bin/bash -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defrag"

[Install]
WantedBy=multi-user.target
user@devazubu227:~$
user@devazubu227:~$
user@devazubu227:~$ sudo systemctl daemon-reexec
user@devazubu227:~$ sudo systemctl daemon-reload
user@devazubu227:~$ sudo systemctl enable disable-thp
Created symlink /etc/systemd/system/multi-user.target.wants/disable-thp.service → /etc/systemd/system/disable-thp.service.
user@devazubu227:~$ sudo systemctl start disable-thp

user@devazubu227:~$ sudo systemctl status disable-thp
○ disable-thp.service - Disable Transparent Huge Pages
Loaded: loaded (/etc/systemd/system/disable-thp.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Thu 2025-08-07 11:01:42 UTC; 11s ago
Process: 4394 ExecStart=/bin/bash -c echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defra>
Main PID: 4394 (code=exited, status=0/SUCCESS)
CPU: 2ms

Aug 07 11:01:42 devazubu227 systemd[1]: Started Disable Transparent Huge Pages.
Aug 07 11:01:42 devazubu227 systemd[1]: disable-thp.service: Deactivated successfully.
user@devazubu227:~$
user@devazubu227:~$
user@devazubu227:~$ cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
user@devazubu227:~$
user@devazubu227:~$ cat /sys/kernel/mm/transparent_hugepage/defrag
always defer defer+madvise madvise [never]
user@devazubu227:~$
user@devazubu227:~$

Splunk Version 9.4.4 shows error while starting - VM CPU Flags are missing

Problem 

When you update your Splunk to e.g. version 9.4.4 and get this error while starting splunk:

Migrating to:
VERSION=9.4.4
BUILD=f627d88b766b
PRODUCT=splunk
PLATFORM=Linux-x86_64

********** BEGIN PREVIEW OF CONFIGURATION FILE MIGRATION **********

-> Currently configured KVSTore database path="/opt/splunk/var/lib/splunk/kvstore"
CPU Vendor: GenuineIntel
CPU Family: 15
CPU Model: 6
CPU Brand: \x
AVX Support: No
SSE4.2 Support: No
AES-NI Support: No

-> isSupportedArchitecture=0
-> isKVstoreDisabled=0
-> isKVstoreDatabaseFolderExist=0
-> isKVstoreDiagnosticsFolderExist=0
-> isKVstoreVersionFileFolderExist=1
-> isKVstoreVersionFileFolderEmpty=0
-> isKVstoreVersionFileMatched=1
-> isKVstoreVersionFromBsonMatched=0
-> isSupportedArchitecture=0
* Active KVStore version upgrade precheck FAILED!
  -- This check is to ensure that KVStore version 4.2 been in use.
  -- In order to fix this failed check, re-install the previous Splunk version, and follow the KVStore upgrade documentation: https://docs.splunk.com/Documentation/Splunk/9.3.0/Admin/MigrateKVstore#Upgrade_KV_store_server_to_version_4.2 .
Some upgrade prechecks failed!
ERROR while running splunk-preinstall.
 

Cause

This might be related to missing CPU features AVX, SSE4.2 and AES-NI to the Splunk VM, which are necessary for the new kvstore mongodb version, which is introduced in Splunk version 9.4: https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/administer-the-app-key-value-store/upgrade-the-kv-store-server-version#Upgrade_the_KV_store_server_version

You can check inside your VM using:
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$ grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u
splnonroot@devubu22h102:/opt/splunk$

Solution 

In your VM hypervisor (VMware ESXi, Microsoft Hyper-V, Proxmox, etc..) --> give the Splunk VMs the necessary CPU flags/features.

Example for proxmox:

  1. Check inside your VM: grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u
  2. Edit /etc/pve/qemu-server/*Your VM ID*.conf and add CPU: Host - so all the Host CPU Hardware flags are forwarded to the VM
  3. Reboot the VM 
  4. Check inside your VM again: grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u
  5. Start Splunk 

Before Proxmox VM has CPU features: 

root@proxmox1:~#
root@proxmox1:~#
root@proxmox1:~# cat /etc/pve/qemu-server/*Your VM ID*.conf
boot: order=scsi0;ide2;net0
cores: 2
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]

[snapshot-pre-splunkupdate]
boot: order=scsi0;ide2;net0
cores: 2
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]
root@proxmox1:~#
root@proxmox1:~# 

After Proxmox VM has CPU features:


1.
root@proxmox1:~#
root@proxmox1:~#
root@proxmox1:~# cat /etc/pve/qemu-server/*Your VM ID*.conf
boot: order=scsi0;ide2;net0
cores: 2
cpu: host
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]

[snapshot-pre-splunkupdate]
boot: order=scsi0;ide2;net0
cores: 2
cpu: host
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]
root@proxmox1:~#
root@proxmox1:~#


2. Reboot the VM

3. Then inside your VM the CPU flags are visible:
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$ grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u
aes
avx
sse4_2
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$ 


4. Start Splunk again

Splunk SearchHead Cluster Artifact Proxying - Splunk internally sharing cached search results

When the same search is run twice in a splunk cluster, is it using a cache for the results or searching the data a second time?

A splunk search head search artifact means the results and metadata from a completed splunk search job (see: https://docs.splunk.com/Splexicon:Searchartifact)

So an artifact is a complete search which is cached for 10minutes.

In a search head cluster the search artifacts are replicated. However this takes a few seconds. What happens, if a search is run again on another search head and the artifact isnt replicated yet to that search head?

The search head captain, which streers those requests uses artifact proxying so the artifact is proxied from the search head which already has completed the search to the other search head.

See also: https://docs.splunk.com/Documentation/Splunk/9.4.0/DistSearch/SHCarchitecture#How_the_cluster_handles_search_artifacts

Example

  • 15 May 2025 11:21:01am - User1 starts the search "index=abc sourcetype=def" @04.May 2025 05:00:00am to 06:00:00am on SH1
  • 15 May 2025 11:21:03am - The search "index=abc sourcetype=def" @04.May 2025 05:00:00am to 06:00:00am on SH1 is complete
  • 15 May 2025 11:21:13am - User2 searches "index=abc sourcetype=def" @04.May 2025 05:00:00am to 06:00:00am on SH2
  • The search head cluster captain will proxy the search artifact (search results) from SH2 to SH1, so the search mustnt run a second time

SPL query for Splunk proxied artifacts

index=_internal host IN (searchhead01*,searchhead02*,searchhead03*) sourcetype=splunkd_access uri_path="/services/search/jobs*" isProxyRequest=true | stats count by method host file

Splunk SearchHead Cluster Artifact Proxying - Splunk internally sharing cached search results


Splunk UseCase for attacks against FortiGate Firewall management interfaces

If you are using Splunk as your SIEM you can try to detect attacks against your FortiGate firewalls by using the following SPL query:


index=firewall type=event subtype=system msg IN ("Failed to match community*", "Message authentication or checking failed*", "Negotiation failed: no matching*", "Negotiation failed: Broken pipe*")
| stats earliest(_time) as FirstEvent count by devname,msg,result,logdesc
| eval FirstEvent=strftime(FirstEvent,"%Y-%m-%d %H:%M:%S")

Splunk UseCase FortiGate Firewall Management Interface Attacks


Additionally: It is imperative that you protect your FortiGate interfaces with TrustedHosts AND Local-In-Policies. Only using TrustHosts protects HTTPS, SSH, etc but not other protocols like SIP, IPsec, CAPWAP, BGP, OSPF, SSLVPN etc which are also local services running on the FortiGate, which need to be protected, too.
See https://how2itsec.blogspot.com/2022/10/fortigate-admin-interface.html

Splunk alert for buckets which are not correctly replicated

The following shows a splunk savedsearch/alert which searches for Splunk buckets which are not correctly replicated to all indexers. 

Example

For example if you have a multisite cluster having 2 sites and each site should contain 2 copies of a bucket: 

splunk_server_clustering_available_sites: "site1,site2"
splunk_server_clustering_site_replication_factor: 'origin:1, site1:2, site2:2, total:4'
splunk_server_clustering_site_search_factor: 'origin:1, site1:2, site2:2, total:4'


Then the following SPL or savedsearch/alert might help identify if multiple buckets of an index are only replicated once:

| dbinspect index=* ```<-- show all buckets of all indexes ``` 
|search NOT state=hot ```<-- only warm & cold buckets ``` 
|eventstats count by bucketId  ```<-- list all bucket-ids only once, count how often they occur ``` 
|search count<2 ```<-- filter for all buckets that occur only once and are not replicated 4 times ``` 
|stats count by index ```<-- show all indexes that have buckets which were replicated only once ``` 
|search count>10 ```<-- show all indexes that have more than 10 buckets which were replicated only once```
``` All buckets should be replicated 4 times according to the search/replication factor of the Splunk multisite cluster. This alert shows if there are indexes with over 10 buckets that are only present once instead of being replicated on 4 indexers``` 


Screenshot:

Splunk bucket only once replicated dbinspect

Explaining screenshot:

Splunk bucket only once replicated dbinspect


Filter logs in Splunk - example filtering monitor probe checks

When running Splunk you want to filter logs, for example to get rid of the many health check probe querys from your monitoring system.

Example filtering PRTG monitoring probe requests using props.conf and transforms.conf

1. Find the monitoring probes in the logs in splunk, e.g.:

10.148.227.111 - - [18/Jul/2024:23:21:06 +0200] "GET /login HTTP/1.1" 200 12882 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:21:06 +0200] "GET / HTTP/1.1" 302 5793 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:20:56 +0200] "GET /login HTTP/1.1" 200 12882 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.111 - - [18/Jul/2024:23:20:56 +0200] "GET / HTTP/1.1" 302 5790 "-" "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)"
10.148.227.121 - - [18/Jul/2024:23:12:17 +0200] "GET /login HTTP/1.1" 200 17480 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com; 576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:16 +0200] "GET / HTTP/1.1" 302 5572 "-" Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com;
576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:15 +0200] "GET /login HTTP/1.1" 200 17486 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com;
576bb8887fe66b1eece876e62e701b9e)"
10.148.227.121 - - [18/Jul/2024:23:12:15 +0200] "GET /login HTTP/1.1" 200 17474 "-" "Mozilla/5.0 (compatible; PaesslerCloudBot/1.0; https://www.paessler.com;
576bb8887fe66b1eece876e62e701b9e)"

2. Create a regex, which finds the logs (which a precise match but as less cpu steps as possible) using https://regex101.com/

regex101.com regex splunk filter

In this example the following regexes where used:

Mozilla\/\d+\.\d+\s+\(compatible;\s+PRTG\s+Network\s+Monitor
Mozilla\/\d.\d\s\(compatible\;\sPaesslerCloudBot\/\d.\d

 

3. Create a dedicated splunk app for this log source or use the default splunk search app and modify the props.conf. Create an entry which you map to the host, source or sourcetype and tell it to use transforms.conf:

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#
uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local# cat props.conf
[...]

#filter prtg monitoring logs
[host::fqdn.of.logsource]
TRANSFORMS-t1=filter-prtg-from-access
TRANSFORMS-t2=filter-prtgcloud-from-access

4. Modify the transforms.conf of this same splunk app. Create an entry which you map to the host, source or sourcetype and force it to the nullQueue:

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#
uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local# cat transforms.conf
#filter prtg logs von access.log von nextcloud
#
[filter-prtg-from-access]
REGEX = Mozilla\/\d.\d\s\(compatible\;\sPRTG\sNetwork\sMonitor
DEST_KEY = queue
FORMAT = nullQueue

[filter-prtgcloud-from-access]
REGEX = Mozilla\/\d.\d\s\(compatible\;\sPaesslerCloudBot\/\d.\d
DEST_KEY = queue
FORMAT = nullQueue

uspunk@ubu2401spl:/opt/splunk/etc/apps/search/local#

5. Reload the splunk configuration using https://your.splunk.fqdn:8000/en-GB/debug/refresh 

6. Your logs should be filtered. If not, check the btool to see if another splunk configuration takes precedence to your configuration:

./splunk btool props list
./splunk btool props list --debug
./splunk btool transforms list
./splunk btool transforms list --debug

Splunk Enterprise update plan

Splunk published this awesome Splunk Enterprise update plan: https://docs.splunk.com/images/d/d3/Splunk_upgrade_order_of_ops.pdf 

Regardless if you have a single-site or multi-site splunk installation, if your are running a stand-alone or distributed and/or clustered architecture, if you are using Splunks Universal Forwarder, the Deployment server, a License Master, Search Head cluster or Indexer Cluster master or not - this plan has your environment setup covered.

Step by step it guides you in updating your Splunk Enterprise environment including backuping up every system, checking each systems health and possible connectivity issues as well as the updates itself, may it be a simple upgrade or a rolling upgrade. Additional informations about each step can be found in the PDF as a link to docs.splunk.com.

Splunk Enterprise update plan step by step


 

Splunk PowerShell SIEM use cases from splunk .conf

Ryan Kovar and Steve Brant from Splunk released on Splunk .conf 2016 a bunch of useful PowerShell SIEM use cases: https://conf.splunk.com/files/2016/slides/hunting-the-known-unknowns-the-powershell-edition.pdf

Finding Un-­encoded IEX Acivity  

Splunk search: sourcetype="WinEventLog:Security" Process_Command_Line=* | evalProcess_Command_Line=lower(Process_Command_Line) | search Process_Command_Line="*iex (new-­‐object net.webclient).downloadstring(*" | stats VALUES(Process_Command_Line) BY host

Screenshot Page 70 from https://conf.splunk.com/files/2016/slides/hunting-the-known-unknowns-the-powershell-edition.pdf

Source: Page 69 and 70 from https://conf.splunk.com/files/2016/slides/hunting-the-known-unknowns-the-powershell-edition.pdf

New Process Started (EventCode 4688)

Splunk search: index=windows source="WinEventLog:Security" (EventCode=4688) NOT (Account_Name=*$) (at.exe OR bcdedit.exe OR chcp.exe OR cmd.exe OR cscript.exe OR ipconfig.exe OR mimikatz.exe OR nbtstat.exe OR nc.exe OR netcat.exe OR netstat.exe OR nmap OR nslookup.exe OR bcp.exe OR sqlcmd.exe OR OSQL.exe OR ping.exe OR powershell.exe OR powercat.ps1 OR psexec.exe OR psexecsvc.exe OR psLoggedOn.exe OR procdump.exe OR rar.exeOR reg.exe OR route.exe OR runas.exe OR sc.exe OR schtasks.exe OR sethc.exe OR ssh.exe OR sysprep.exe OR systeminfo.exe OR system32\\net.exe OR tracert.exe OR vssadmin.exe OR whoami.exe OR winrar.exe OR wscript.exe OR winrm.* OR winrs.* OR wmic.exe OR wsmprovhost.exe) | evalMessage=split(Message,".") | evalShort_Message=mvindex(Message,0) | table _Ome, host, Account_Name, Process_Name, Process_ID, Process_Command_Line, New_Process_Name, New_Process_ID, Creator_Process_ID, Short_Message

Page 83 from https://conf.splunk.com/files/2016/slides/hunting-the-known-unknowns-the-powershell-edition.pdf
 

Source: Page 82 and 83 from https://conf.splunk.com/files/2016/slides/hunting-the-known-unknowns-the-powershell-edition.pdf

Finding Modules (EventCode 4103 or 4104)  

Splunk search: sourcetype="WinEventLog:Microsoft-Windows-PowerShell/Operational" (EventCode=4104) OR (EventCode=4103)(Set-ExecutionPolicyOR Set-MasterBootRecordl OR Get-WMIObject OR Get-GPPPassword OR Get-Keystrokes OR Get-TimedScreenshot OR Get-VaultCredential OR GetServiceUnquoted OR Get-ServiceEXEPerms OR Get-ServicePerms OR Get-RegAlwaysInstallElevated OR Get-RegAutoLogon OR Get-UnattendedInstallFiles OR Get-Webconfig OR Get-ApplicationHost OR Get-PassHashes OR Get-LsaSecret OR GetInformation OR Get-PSADForestInfo OR Get-KerberosPolicy OR Get-PSADForestKRBTGTInfo OR Get-PSADForestInfo OR GetKerberosPolicy OR Invoke-Command OR Invoke-Expression OR iex OR Invoke-Shellcode OR Invoke--Shellcode OR Invoke-ShellcodeMSIL OR InvokeMimikatzWDigestDowngrade OR Invoke-NinjaCopy OR Invoke-CredentialInjection OR Invoke-TokenManipulation OR InvokeCallbackIEX OR Invoke-PSInject OR Invoke-DllEncode OR Invoke-ServiceUserAdd OR Invoke-ServiceCMDOR Invoke-ServiceStart OR Invoke-ServiceStop OR Invoke-ServiceEnable OR Invoke-ServiceDisable OR Invoke-FindDLLHijack OR Invoke-FindPathHijack OR Invoke-AllChecks OR Invoke-MassCommand OR Invoke-MassMimikatz OR Invoke-MassSearch OR Invoke-MassTemplate OR Invoke-MassTokens OR Invoke-ADSBackdoor OR Invoke-CredentialsPhish OR Invoke-BruteForce OR Invoke-PowerShellIcmp OR Invoke-PowerShellUdp OR Invoke-PsGcatAgent OR Invoke-PoshRatHttps OR Invoke-PowerShellTcp OR Invoke-PoshRatHttp OR Invoke-PowerShellWmi OR Invoke-PSGcat OR Invoke-Encode OR Invoke-Decode OR Invoke-CreateCertificate OR InvokeNetworkRelay OR EncodedCommand OR New-ElevatedPersistenceOption OR wsman OR Enter-PSSession OR DownloadString OR DownloadFile OR Out-Word OR Out-Excel OR Out-Java OR Out-Shortcut OR Out-CHM OR Out-HTA OR Out-Minidump OR HTTP-Backdoor OR FindAVSignature OR DllInjection OR ReflectivePEInjection OR Base64 OR System.Reflection OR System.Management OR Restore-ServiceEXE OR Add-ScrnSaveBackdoor OR Gupt-Backdoor OR Execute-OnTime OR DNS_TXT_Pwnage OR WriteUserAddServiceBinary OR Write-CMDServiceBinary OR Write-UserAddMSI OR Write-ServiceEXE OR Write-ServiceEXECMD OR Enable-DuplicateToken  OR Remove-Update OR Execute-DNSTXT-Code OR Download-Execute-PS OR Execute-CommandMSSQL OR Download_Execute OR Copy-VSS OR Check-VM OR Create-MultipleSessions OR Run-EXEonRemote OR Port-Scan OR Remove-PoshRat OR TexttoEXE OR Base64ToString OR StringtoBase64 OR Do-Exfiltration OR Parse_Keys OR Add-Exfiltration OR AddPersistence OR Remove-Persistence OR Find-PSServiceAccounts OR Discover-PSMSSQLServers OR DiscoverPSMSExchangeServers OR Discover-PSInterestingServices OR Discover-PSMSExchangeServers OR DiscoverPSInterestingServices OR Mimikatz OR powercat OR powersploit OR PowershellEmpire OR Payload OR GetProcAddress) 

Page 85 from https://conf.splunk.com/files/2016/slides/hunting-the-known-unknowns-the-powershell-edition.pdf

Source: Page 84 and 85 from https://conf.splunk.com/files/2016/slides/hunting-the-known-unknowns-the-powershell-edition.pdf

SIEM Use Case - find suspicious powershell commands

Microsofts Powershell is a very mighty tool, which can be used as LoLBin. To detect suspicious powershell commands or scripts, a SIEM use case in order to find suspicious powershell-commands can be:

Logging / Data Source

Active PowerShell Script Block Logging (Event ID 4104) OR use your Advanced Endpoint Protection AEP or Endpoint Detection and Response EDR tool like VMware Carbon Black, Microsoft Defender ATP, Crowdstrike or the other tools.

SIEM use case / fetch suspicious powershell

1. process = powershell.exe

&&

2. cmd = ToBase64String OR FromBase64String OR -e OR -en OR -enc OR -enco OR -encod OR -encode OR -encoded OR -encodedc OR -encodedco OR -encodedcom OR -encodedcomm OR -encodedcomma OR -encodedcomman OR -encodedcommand OR -ec

&&

3. not cmd = Windows\CCM\*

More very useful information

SIEM IoC regsvr32.exe outbound network connection

An easy to find possible indicator of compromise (IoC) for your SIEM, AEP or EDR could be a outbound network connection from Windows own register server regsvr32.exe (Microsoft Docs or Wiki). Normally the register server never establishes an outbound network connection to the internet. It is a commonly used evasion technique to avoid detection and has its own MITRE Att&ck technique with ID T1117 (or new sub-techniques T1218/010 and can be mapped to the MITRE Att&ck tactics Execution TA0002 and Defense Evasion TA0005.

A starting point can be searching your SIEM logs for network connections from regsvr32.exe to a not RFC1918 private ip address and your IPv6 address space.

Mitigations could be using the Windows firewall to block outbound network connections from regsvr32.exe or as MITRE Att&ck writes:

"Microsoft's Enhanced Mitigation Experience Toolkit (EMET) Attack Surface Reduction (ASR) feature can be used to block regsvr32.exe from being used to bypass whitelisting. Identify and block potentially malicious software executed through regsvr32 functionality by using application whitelisting tools, like Windows Defender Application Control, AppLocker, or Software Restriction Policies where appropriate."

More useful searches for Splunk & Sysmon environments can be found on Github, example: https://github.com/mitre-attack/car/issues/11 and testing if your AEP/EDR/Sysmon or log-collection-tool actually logs regsvr32 events is described here: https://github.com/redcanaryco/atomic-red-team/blob/master/atomics/T1117/T1117.md
 

Important Splunk update for timestamp issue starting 01.01.2020

Splunk has identified an issue regarding timestamps, which are intensivly used for data correlation in many Splunkbased logging and SIEM systems. Affected versions are Splunk Enterprise, Splunk Light and Splunk Cloud. This issue has potential significant impact on data ingestion - including causing inaccurate, unsearchable, or prematurely-deleted data - starting January 1, 2020.

Cause of the issue: Timestamps using two-digit years will stop being correctly recognized. Full details around this issue, including workarounds and product fixes, are documented in Release Notes for each Splunk Version: https://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/FixDatetimexml2020
 

Possible Solutions:

1. Manual Change of "datetimes.xml"

Change on all Splunk systems (search heads, indexers, heavy forwards etc) the file "datetimes.xml" with the following file: http://download.splunk.com/products/ingest2020/datetime.zip
In order to do that, put the downloaded in $SPLUNK_HOME/etc. (mostly found in /opt/splunk/etc). Then restart the system (in a Indexer Cluster a rolling-restart is possible). Until a splunk patch is available, the warning will be shown, that this file is not part of the splunk manifest. This will be fixed in the future splunk versions.

2. Update to a version with a fix

Splunk will ship minor releases with fixes, soon:

Major Release --- Minor Release with patch
6.6 --- 6.6.12.1 (not yet released)
7.0 --- 7.0.13.1 (not yet released)
7.1 --- 7.1.10 (not yet released)
7.2 --- 7.2.9.1 (not yet released)
7.3 --- 7.3.3 (Installationguide)
8.0 --- 8.0.1 (not yet released)

Proxmox Intel NUC crashes - Detected Hardware Unit Hang

Problem  If you are running proxmox (e.g. proxmox v 8.4.14) on a Intel NUC and it sporadically crashes with the following entries in the log...