Veeam backup causes BGP route flapping on VMware NSX-T Edge VMs

When running VMware NSX-T with BGP and BFD and you are using Veeam backup, you may see BGP route flapping or BGP neighbor adjchanges or Down BGP Notification FSM-ERR.

Issue could be caused by Veeam backup, which is creating a snapshot of your NSX-T edge VM in order to back it up.

Logs show something like:
2020-12-20T20:38:05.278Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 142898 us
2020-12-20T20:35:05.806Z| vcpu-0| I125: SnapshotVMXTakeSnapshotComplete: Done with snapshot 'VEEAM BACKUP TEMPORARY SNAPSHOT': 153

Router logs show something like:
date=2020-12-20,time=20:35:10,devname="fwdev01",logid="0103020300",type="event",subtype="router",level="warning",vd="dev",eventtime=693310,logdesc="BGP neighbor status changed",msg="BGP: %BGP-5-ADJCHANGE: neighbor 172.23.39.35 Up "
date=2020-12-20,time=20:35:10,devname="fwdev01",logid="0103020301",type="event",subtype="router",level="warning",vd="dev",eventtime=693310,logdesc="Routing log",msg="BGP: 172.23.39.35-Outgoing [DECODE] Open Cap: unrecognized capability code 73 len 8"
date=2020-12-20,time=20:35:10,devname="fwdev01",logid="0103020301",type="event",subtype="router",level="warning",vd="dev",eventtime=693310,logdesc="Routing log",msg="BGP: 172.23.39.35-Outgoing [DECODE] Open Cap: unrecognized capability code 69 len 4"
date=2020-12-20,time=20:35:06,devname="fwdev01",logid="0103020300",type="event",subtype="router",level="warning",vd="dev",eventtime=693306,logdesc="BGP neighbor status changed",msg="BGP: %BGP-5-ADJCHANGE: neighbor 172.23.39.35 Down BGP Notification FSM-ERR"
date=2020-12-20,time=20:35:06,devname="fwdev01",logid="0103020301",type="event",subtype="router",level="warning",vd="dev",eventtime=693306,logdesc="Routing log",msg="BGP: %BGP-3-NOTIFICATION: received from 172.23.39.35 6/2 (Cease/Administratively Shutdown.) 0 data-bytes

Hint when VMWare doesnt show a snapshot

If VMWare ESXi/vCenter/your vSphere environment does not show a virtual machine snapshot, however it is there and the virtual machine is using it, here is a little trick which might help you:

1. Make sure you have some free space left in your datastore, in which the vm is stored.
2. Create another virtual machine snapshot 
3. Click on "delete all snapshots"

This can trigger the ESXi to delete all snapshots, including the one which isn't shown. However this does not always work, in which case working with a clone of the VM and vmkfstools can help.