Issue with Virtual IPs and NAT

[bugleboy]

I'm currently setting up a firewall pair with HA and Multi-WAN configurations using the OPNsense documentation. I'm having an issue on both firewalls with one WAN interface. (i've disabled any HA and shut off the 2nd firewall for the time being, so this is essentially a typical multi-wan set up ATM). When an outgoing NAT rule is set to use WAN1's VIP, I cannot connect to/ping anything. A packet analysis shows that the packets are going out through the firewall, but never receiving a response. My gateway/firewall can still successfully ping out, though.

I thought it was an issue with my NAT rule, but it's set up the same as my WAN2 rule which functions properly. I have "Automatic outbound NAT for Reflection" checked, and am using sticky connections to help rule out any issues with my ISP. I also confirmed that both addresses given to me from my ISP work fine, just not when they're set to be used as the VIP/Outgoing NAT. I have floating firewall rules to allow CARP on all interfaces, and I haven't found anything different between my WAN1 and WAN2 configurations yet.

Any help would be appreciated

[Monviech (Cedrik)]

Maybe its a MAC learning issue of the gateway that routes these IPs to you.

You do proxy arp for additional IP addresses on the same interface, maybe a device does not learn these IPs.

If you cannot ping them directly from the internet or do a traceroute and these IPs are not the final hop its a hint.

[bugleboy]

It seems like it was something like that. I switched to having the VIP's MAC be used by my physical WAN interfaces, and things are working somewhat smoothly right now. I am seeing intermittent 8-11% packetloss on each firewall's WAN GW, though. I'm assuming this because of my switches having to relearn paths and such. I called my ISP and they said they don't block anything like that and others with my use case have encountered no issues. My second ISP on WAN2 has had no issues dealing with this, though.

[Fabian Wenk]

With CARP the Virtual IP does have its own dedicated MAC address. With this a upstream router does not have to learn a new MAC address when the VIP does jump from one OPNsense instance to the other.
But on the other hand now the switch between the OPNsense and the upstream router does have to reassign the MAC address to another port. Depending on possible security settings on the switch, this may not work properly.