18.7.10_3 Loses interface

[dragon2611]

upgraded a Pair with one physical opnsense and one virtual from 18.7.6 i think it was to 18.7.10_3 and the now one of my Vlans can't ping out via the physical unit

its bce0_vlan101 so a broadcom NIC.

Oddly when I put the physical machine into carp maintenance mode and rebooted it I was able to ping it's IP address from one of the VM's on that Vlan, as soon as I took it out of maintenance mode and it took over the VIP I lost the ability for the VM to ping both the firewalls Real IP and the virtual ip.

At the moment I've left it in maintenance mode with the virtual secondary handling the traffic, one difference is the VM doesn't have Vlans where as the psychical does for the VM the Vlan tagging is done by the hypervisor so opt1,opt2.etc is just seen by opnsense as an additional nic

[mimugmail]

You are sure it was running with 18.7.6?
Can you check if opt1, opt2 etc are assigned on both machines same way (e.g. DMZ = opt2, WAN2 = opt3)
Check system.log for errors when putting out of maintenance mode.

[dragon2611]

not 100% sure, I know it was an 18.7 release before I updated and it also hadn't been done for a while.

The interfaces appear to be assigned ok, I'll reboot and do some further testing and see if I can come back with something slightly more useful than "it's broke".

Edit:

Looks like I ran into this issue - https://github.com/zerotier/ZeroTierOne/issues/787 

The subnet on bce0_vlan101 was advertised as a managed route in zerotier pointing at the VIP that my firewalls have on that Zerotier interface instead of the expected connected route for the /24 opnsense was learning the managed route from ZT in preference to the connected route.  Effectively giving it a route for that /24 that points at itself 

Now why it only affected one of the firewalls and not the other one I don't have a clue as they are both connected to Zerotier, also not sure why it wasn't a problem before now either but whatever, least I've gotten to the bottom of it.

I've removed the managed route from zerotier.com for now, but ideally the Opnsense plugin for zerotier needs the option to ignore managed routes from zerotier or an interface to the blacklist config.