2 WAN Uplinks split routing issues with incoming connections

[paul5012]

Hi,

I've got a problem with a setup as in the drawing:
2 Internet uplinks, each one has a FritzBox. Different providers, each one has a static IPv4 address.
Only FritzBox2 has IPv6 (static address + /56 static prefix, but not of interest here)
OPNsense is 26.4.1-amd64

What I want to have: load balanced WAN links. Services in the DMZ like a Nextcloud or a mail gateway should be reachable via both public (IPv4) adresses.

I followed the intructions in https://docs.opnsense.org/manual/how-tos/multiwan.html and did multiple searches but found no solution.

The gateways table has 3 entries, two for the IPV4 Fritzboxes and one for the IPv6 box. Both v4 gateways have the same priority of 63. I configured a monitor IP (1.1.1.1 and 1.0.0.1 on the other interface).
The gateway is present in the respective WAN interface definition.
There is a gateway group with both v4 gateways, both as tier 1.
Pool options "default", trigger level "packet loss and high latency"

I did not configure DNS servers for each gateway as want unbound to be a full recursor in did not get the point with this part of the story.
I modified the "LAN pass to all" rule as in Step 4.

In the gateway overview one of the Fritzboxes is labeled "active", and there goes all the traffic.
When I try to connect from the internet to the nginx reverse proxy, I succeed when using the address of the "active" Fritzbox.
When I try to access the other public IP the packets are natted from the Fritzbox correctly and the syn packet arrives at the OPNsense. But the syn-ack packet go the wrong interface, with the sender address of the interface where to syn came in.

"Use sticky connections" is on. "shared forwarding" and "Disable force gateway" are off.

What do I miss?

[pfry]

Sounds like you want a policy routing solution (""Gateway" in the rule definitions) for incoming sessions. You'd need to differentiate your policies by destination address. Load-sharing outbound? Got me.

(I'd normally use VRFs for something like this, but hey.)

[paul5012]

would I find something in the documentation, how to achieve this?

[pfry]

Documentation is pretty light... I don't know of any examples. Searching this forum would probably be your best bet for that.

[viragomann]

Is the gateway status shown up as "online" for both IPv4 gateways in System: Gateways: Configuration?

How did you configure firewall rule for incoming traffic?