Strange Issue - IP missing from interface & can't update it; Showing virtual IP

[e]

Hi all,

I have been running into a very odd problem for a little while now and am unable to find any help anywhere else online. This started after a power outage which UPS was unable to hold over. When everything was brought back online, the entire network segment on the main client/server VLAN (vlan0.10) was offline and I could not reach the OPNsense server via Virtual IP or the interface Static IP. Looking at the console on the physical server, every other VLAN interface has the correct interface Static IP except vlan0.10 which was listing the Virtual IP for CARP.

I was able to connect to the web GUI on another MGMT interface, and going to that interface settings, it is showing the correct Static IP configured, but still shows the Virtual IP for the interface in the console. I also cannot access the server via either of those IPs on that interface. Also checking the ifconfig output, every VLAN interface has both the interface static IP as well as the Virtual IP if applicable, while vlan0.10 only has either the single Virtual IP or no inet configuration depending on as follows.

I have tried the following:
- Disabling/re-enabling the interface: no change
- Rebooting the server: no change
- Setting the IP address to None, then Static: in the console the IP is now blank, no change when set back to static
- Changing the IP address: no change
- Changing the netmask: console now shows the IP/netmask that was set but still no connectivity
- Changing netmask back to /24: reverts back to previous Virtual IP
- Setting Interface IP from console: doesn't apply changes
- Changing between Persistent Maintenance Mode  in CARP settings: no change
- Disabling CARP: no change

I ended up giving up and powered down the server, and ran just the Backup instance (virtual machine) for a few days. When I circled back around to this and restarted the Primary server, it came back up with the correct IPs without issue. Figured I was all set at this point and was fine for a few weeks. Now just recently I installed firmware update and upon rebooting, I lost connection to the server again, and lo and behold, the same this is happening once again.

I did more troubleshooting steps outlined above and am still failing to get the Static IP for this interface to apply. When the network is running happily on the Backup VM instance, as soon as the Primary server boots, I loose the ping to the Virtual IP and the entire VLAN drops offline. I gave up again and powered the server off.


Basic network setup:

Primary OPNsense: SuperMicro server bare metal
Secondary OPNsense: VM on ProxMox host

CARP Virtual IP for vlan0.10: 10.0.10.1/24
Primary Static IP for vlan0.10: 10.0.10.2/24
Secondary Static IP for vlan0.10: 10.0.10.3/24

Primary has 2 NICs configured as LAGG to Brocade switch, and all VLAN traffic including WAN, LAN, PFSYNC, and additional VLANS is tagged. VLAN interfaces are bridged in OPNsense from the LAGG interface. Secondary takes the VLAN aware bridge interface from Proxmox (also a LAGG within Proxmox) and gets the same tagged VLANs.

I will be able to post configs or any other info that could help later when I am able to power up Primary once again. Everything was working fine previously including CARP fail-overs and rebooting instances with no ill effects and minimal packet loss. My next step would be fresh OPNsense install on the server and restore from config backup, but I am asking here first.

Has anybody ever ran into anything vaguely similar to this before and can provide some assistance?


Many Thanks,
e