Problem
When you update your Splunk to e.g. version 9.4.4 and get this error while starting splunk:
Migrating to:
VERSION=9.4.4
BUILD=f627d88b766b
PRODUCT=splunk
PLATFORM=Linux-x86_64
********** BEGIN PREVIEW OF CONFIGURATION FILE MIGRATION **********
-> Currently configured KVSTore database path="/opt/splunk/var/lib/splunk/kvstore"
CPU Vendor: GenuineIntel
CPU Family: 15
CPU Model: 6
CPU Brand: \x
AVX Support: No
SSE4.2 Support: No
AES-NI Support: No
-> isSupportedArchitecture=0
-> isKVstoreDisabled=0
-> isKVstoreDatabaseFolderExist=0
-> isKVstoreDiagnosticsFolderExist=0
-> isKVstoreVersionFileFolderExist=1
-> isKVstoreVersionFileFolderEmpty=0
-> isKVstoreVersionFileMatched=1
-> isKVstoreVersionFromBsonMatched=0
-> isSupportedArchitecture=0
* Active KVStore version upgrade precheck FAILED!
-- This check is to ensure that KVStore version 4.2 been in use.
-- In order to fix this failed check, re-install the previous Splunk version, and follow the KVStore upgrade documentation: https://docs.splunk.com/Documentation/Splunk/9.3.0/Admin/MigrateKVstore#Upgrade_KV_store_server_to_version_4.2 .
Some upgrade prechecks failed!
ERROR while running splunk-preinstall.
Cause
This might be related to missing CPU featuresAVX, SSE4.2 and AES-NI to the Splunk VM, which are necessary for the new kvstore mongodb version, which is introduced in Splunk version 9.4: https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/9.4/administer-the-app-key-value-store/upgrade-the-kv-store-server-version#Upgrade_the_KV_store_server_versionYou can check inside your VM using:
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$ grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u
splnonroot@devubu22h102:/opt/splunk$
Solution
In your VM hypervisor (VMware ESXi, Microsoft Hyper-V, Proxmox, etc..) --> give the Splunk VMs the necessary CPU flags/features.Example for proxmox:
- Check inside your VM:
grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u - Edit
/etc/pve/qemu-server/*Your VM ID*.confand addCPU: Host- so all the Host CPU Hardware flags are forwarded to the VM - Reboot the VM
- Check inside your VM again:
grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u - Start Splunk
Before Proxmox VM has CPU features:
root@proxmox1:~#
root@proxmox1:~#
root@proxmox1:~# cat /etc/pve/qemu-server/*Your VM ID*.conf
boot: order=scsi0;ide2;net0
cores: 2
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]
[snapshot-pre-splunkupdate]
boot: order=scsi0;ide2;net0
cores: 2
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]
root@proxmox1:~#
root@proxmox1:~# After Proxmox VM has CPU features:
1.
root@proxmox1:~#
root@proxmox1:~#
root@proxmox1:~# cat /etc/pve/qemu-server/*Your VM ID*.conf
boot: order=scsi0;ide2;net0
cores: 2
cpu: host
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]
[snapshot-pre-splunkupdate]
boot: order=scsi0;ide2;net0
cores: 2
cpu: host
ide2: local:iso/ubuntu-22.04-live-server-amd64_2.iso,media=cdrom
memory: 8192
name: devubu22h102
net0: virtio=CA:*redacted*:CC,bridge=vmbr0,firewall=1
[...]
root@proxmox1:~#
root@proxmox1:~#2. Reboot the VM
3. Then inside your VM the CPU flags are visible:
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$ grep -o -w 'sse4_2\|avx\|aes' /proc/cpuinfo | sort -u
aes
avx
sse4_2
splnonroot@devubu22h102:/opt/splunk$
splnonroot@devubu22h102:/opt/splunk$ 4. Start Splunk again
