Veeam backup causes BGP route flapping on VMware NSX-T Edge VMs

When running VMware NSX-T with BGP and BFD and you are using Veeam backup, you may see BGP route flapping or BGP neighbor adjchanges or Down BGP Notification FSM-ERR.

Issue could be caused by Veeam backup, which is creating a snapshot of your NSX-T edge VM in order to back it up.

Logs show something like:
2020-12-20T20:38:05.278Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 142898 us
2020-12-20T20:35:05.806Z| vcpu-0| I125: SnapshotVMXTakeSnapshotComplete: Done with snapshot 'VEEAM BACKUP TEMPORARY SNAPSHOT': 153

Router logs show something like:
date=2020-12-20,time=20:35:10,devname="fwdev01",logid="0103020300",type="event",subtype="router",level="warning",vd="dev",eventtime=693310,logdesc="BGP neighbor status changed",msg="BGP: %BGP-5-ADJCHANGE: neighbor 172.23.39.35 Up "
date=2020-12-20,time=20:35:10,devname="fwdev01",logid="0103020301",type="event",subtype="router",level="warning",vd="dev",eventtime=693310,logdesc="Routing log",msg="BGP: 172.23.39.35-Outgoing [DECODE] Open Cap: unrecognized capability code 73 len 8"
date=2020-12-20,time=20:35:10,devname="fwdev01",logid="0103020301",type="event",subtype="router",level="warning",vd="dev",eventtime=693310,logdesc="Routing log",msg="BGP: 172.23.39.35-Outgoing [DECODE] Open Cap: unrecognized capability code 69 len 4"
date=2020-12-20,time=20:35:06,devname="fwdev01",logid="0103020300",type="event",subtype="router",level="warning",vd="dev",eventtime=693306,logdesc="BGP neighbor status changed",msg="BGP: %BGP-5-ADJCHANGE: neighbor 172.23.39.35 Down BGP Notification FSM-ERR"
date=2020-12-20,time=20:35:06,devname="fwdev01",logid="0103020301",type="event",subtype="router",level="warning",vd="dev",eventtime=693306,logdesc="Routing log",msg="BGP: %BGP-3-NOTIFICATION: received from 172.23.39.35 6/2 (Cease/Administratively Shutdown.) 0 data-bytes

Intel NUC 10th gen running VMware ESXi 7.0

Due to growing data I had to add more storage. Therefore I bought a new Intel NUC (10th generation)  running VMware ESXi 7.0. Really helpful for the setup is again virten.net, which provides all the necessary information.

Simply installing the ESXi on the NUC using a USB stick. For creating the USB stick I used rufus. For the ESXi image use the steps from virten.net, in order to create an ESXi 7.0 image with a network interface card driver which works for Intel NUC 10th gen (otherwise an error about "No Network Adapters" is shown).

Start PowerShell (with Admin-Rights) and type in:

Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
Export-ESXImageProfile -ImageProfile "ESXi-7.0.0-15843807-standard" -ExportToBundle -filepath ESXi-7.0.0-15843807-standard.zip
Remove-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
Add-EsxSoftwareDepot .\ESXi-7.0.0-15843807-standard.zip
Add-EsxSoftwareDepot .\ESXi670-NE1000-32543355-offline_bundle-15486963.zip
New-EsxImageProfile -CloneProfile "ESXi-7.0.0-15843807-standard" -name "ESXi-7.0.0-15843807-NUC" -Vendor "virten.net"
Remove-EsxSoftwarePackage -ImageProfile "ESXi-7.0.0-15843807-NUC" -SoftwarePackage "ne1000"
Add-EsxSoftwarePackage -ImageProfile "ESXi-7.0.0-15843807-NUC" -SoftwarePackage "ne1000 0.8.4-3vmw.670.3.99.32543355"
Export-ESXImageProfile -ImageProfile "ESXi-7.0.0-15843807-NUC" -ExportToIso -filepath ESXi-7.0.0-15843807-NUC.iso
Export-ESXImageProfile -ImageProfile "ESXi-7.0.0-15843807-NUC" -ExportToBundle -filepath ESXi-7.0.0-15843807-NUC.zip

If there is an issue "about_Execution_Policies" (https:/go.microsoft.com/fwlink/?LinkID=135170), like:

+ Import-Module VMware.ImageBuilder
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : Sicherheitsfehler: (:) [Import-Module], PSSecurityException
    + FullyQualifiedErrorId : UnauthorizedAccess,Microsoft.PowerShell.Commands.ImportModuleCommand
 

then you can help yourself using the following temporary workaround:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned

❗Warning! This is a possible security issue (see MS documentation). Set this setting back to default after creating the image using:

Set-ExecutionPolicy -ExecutionPolicy Default 

Update 04.01.2021: After having problems with large file transfers from and to the ESXi or from and to VMs running on the ESXi, I've reinstalled ESXi6.7u3 on the NUC. The problems continued and large file transfers using SCP, using SFTP or HTTPS always were corrupted or broke up, no matter which application or operating system. So I decided to switch to proxmox. Proxmox and the VMs on Proxmox work fine and have no issues.

Intel NUC running VMware ESXi 6.5.3

Because there are some people on the internet asking for experience on "is my Intel NUC supporting ESXi version x.y.z?", so I will provide short feedback regarding that: Using an Intel NUC NUC7i7DNHE "Intel NUC 8th Gen Commercial" (https://www.intel.de/content/www/de/de/products/boards-kits/nuc.html) I'm running VMware ESXi 6.5 update 3 without any additional drivers necessary.

VMware updates can be found here: http://www.vmware.com/patchmgr/download.portal

Hint when VMWare doesnt show a snapshot

If VMWare ESXi/vCenter/your vSphere environment does not show a virtual machine snapshot, however it is there and the virtual machine is using it, here is a little trick which might help you:

1. Make sure you have some free space left in your datastore, in which the vm is stored.
2. Create another virtual machine snapshot 
3. Click on "delete all snapshots"

This can trigger the ESXi to delete all snapshots, including the one which isn't shown. However this does not always work, in which case working with a clone of the VM and vmkfstools can help.