VLANs, HA/Carp, Unbound and now MultiWAN - the latter not working with VLANs

[Shihatsu]

So my Setup was basically this:
Fritzbox with DSL.
Two OPNsense connectect to the Fritzbox, both in CARP HA setup - working fine:
https://www.thomas-krenn.com/en/wiki/OPNsense_HA_Cluster_configuration
A Mikrotik Switch connected to the OPNsenses and everything else, including WiFi-AP.
The OPNsense had 12 networks with the repective interfaces: LAN, WAN, CARP, VLAN10,20,...90.
I have Unbound DNS running for my naming solution.
Until here everything is working fine.

Now I added a seceond WAN port on OPNsense 1, which stands for a second Gateway - MultiWAN setup to get additional Redundancy - have a mayor incidence with my DSL provider. I used the following guide mostly:
https://docs.opnsense.org/manual/how-tos/multiwan.html
It is working fine for my LAN, but all my VLANs are not working. I have done the following:

• Add the second gateway and make it work - when I did this I did not check the VLAN functionality. I assume it is not working. But LAN is working fine, so the gateway is doing its job (It is a Fritzbox 6820LTE, but will be replaced soon). In the same step I added monitoring IPs (DNS of quad9) and unchecked "Disable Gateway Monitoring" and "Mark Gateway as Down".
• Creating a gateway group, adding trigger, priority and tiers
• Changing the "Default Access to all rule" of LAN and a first VLAN to Gateway group from *
• I did not change DNS under System ‣ Settings ‣ General because I use Unbound (tried this ofc, does not fix my error)
• I allready had a DNS allow rule, because I also have a "disallow external DNS" (to avoid devices use not Unbound)

So, after I ahve done all of this I can access the Internet and everything in Lan, but nothing in VLAN. What do I miss, where to look? I can ping the VLANs from the respective OPNsense (Interface Diagnosis). I cannot access the respective "other" Fritzbox, only the active one which is connected to my current Gateway.

[Shihatsu]

For naybody who may come into the same problem: the firewall rules (the access to all rule) need to be changed to gatewaygroup, but a rule has to be added also - and this one does not use "gateway group" as gateway, but the standard asterix, and this one must towar all VLANs. Me is dumb, thank you.

[Skaamy]

Hello,

I've a similar config, no issue with VLANs but with CARP/HA. Can you tell me if it behave the same for you ?

Config is :
- 2 WAN with a gateway group
- CARP configured for WAN/LAN

When my first WAN goes down, it does no use the second one, but CARP is taking the advantage and all my traffic is handled by the second OPNSense.

Thanks in advance if you can tell me how it works for you...

[Shihatsu]

Hello,

I've a similar config, no issue with VLANs but with CARP/HA. Can you tell me if it behave the same for you ?

Config is :
- 2 WAN with a gateway group
- CARP configured for WAN/LAN

When my first WAN goes down, it does no use the second one, but CARP is taking the advantage and all my traffic is handled by the second OPNSense.

Thanks in advance if you can tell me how it works for you...

Heya, a fellow painstaker... Yeah, behaves the same, and I think it is the "correct" behaviour. I will tinker a bit with it later on and will share my information, my idea is the following: CARP1 has WAN1 als T1 and WAN2 as T2 and CARP2 has WAN2 as T1 and WAN1 as T2.
I guess MultiWAN and CARP need to be updated, because this scenario is highly undocumented.

[Skaamy]

Hi !
OK, let me know if you find something...
Maybe the trick is to not configure the CARP IP with the public IP address, but with a private one.
And not use transparent mode for WAN interface but router mode (ISP side).
Like that, even if the public IP address does not answer anymore, the CARP will not detect the failure, because the WAN interface private IP will still answer...

Need to be tested ! I'll do when i have some spare time...