2 WAN Uplinks split routing issues with incoming connections

[paul5012]

Hi,

I've got a problem with a setup as in the drawing:
2 Internet uplinks, each one has a FritzBox. Different providers, each one has a static IPv4 address.
Only FritzBox2 has IPv6 (static address + /56 static prefix, but not of interest here)
OPNsense is 26.4.1-amd64

What I want to have: load balanced WAN links. Services in the DMZ like a Nextcloud or a mail gateway should be reachable via both public (IPv4) adresses.

I followed the intructions in https://docs.opnsense.org/manual/how-tos/multiwan.html and did multiple searches but found no solution.

The gateways table has 3 entries, two for the IPV4 Fritzboxes and one for the IPv6 box. Both v4 gateways have the same priority of 63. I configured a monitor IP (1.1.1.1 and 1.0.0.1 on the other interface).
The gateway is present in the respective WAN interface definition.
There is a gateway group with both v4 gateways, both as tier 1.
Pool options "default", trigger level "packet loss and high latency"

I did not configure DNS servers for each gateway as want unbound to be a full recursor in did not get the point with this part of the story.
I modified the "LAN pass to all" rule as in Step 4.

In the gateway overview one of the Fritzboxes is labeled "active", and there goes all the traffic.
When I try to connect from the internet to the nginx reverse proxy, I succeed when using the address of the "active" Fritzbox.
When I try to access the other public IP the packets are natted from the Fritzbox correctly and the syn packet arrives at the OPNsense. But the syn-ack packet go the wrong interface, with the sender address of the interface where to syn came in.

"Use sticky connections" is on. "shared forwarding" and "Disable force gateway" are off.

What do I miss?

[pfry]

Sounds like you want a policy routing solution (""Gateway" in the rule definitions) for incoming sessions. You'd need to differentiate your policies by destination address. Load-sharing outbound? Got me.

(I'd normally use VRFs for something like this, but hey.)

[paul5012]

would I find something in the documentation, how to achieve this?

[pfry]

Documentation is pretty light... I don't know of any examples. Searching this forum would probably be your best bet for that.

[viragomann]

Is the gateway status shown up as "online" for both IPv4 gateways in System: Gateways: Configuration?

How did you configure firewall rule for incoming traffic?

[wincent]

I thank you need a layer 3 switch that supports PBR to be placed between two FritzBoxes and OPNsense devices.

[paul5012]

yes, both gateways are show as "online".

the firewall rule for incoming traffic is one NAT rule per port, which includes 2 ports (or sometimes even 4, if I want for notebooks or mobile phones to work from the outside (internet) as from the inside (Wifi bridged to LAN) with the same configuration).
I also used the firewall - settings - advanced - Network address translation settings "Reflection for destination NAT" and "Automatic outbound NAT for Reflection".
That seems to work great to get my mobile devices work from in- and outside with only few explicit rules.
But it gives me trouble an another point, and I'm not quite sure how to overrule these settings in an individual rule.
But this is just the next thread.

In the NAT rules I set the "Firewall rule" to "Pass"

[wincent]

This is not a firewall rule issue, it's a routing issue, The default route gateway of the firewall is only one. If the inbound traffic happens to be on your default circuit, there is no problem. If the inbound traffic is on another circuit, the firewall's outbound packets cannot reach this route. You need a layer 3 switch to run PBR to handle this.

[pfry]

[...]You need a layer 3 switch to run PBR to handle this.

? The firewall can handle this under most circumstances.

[wincent]

The firewall only handles the outbound load balancing, for inbound traffic, if PBR is not deployed in your front-end, the firewall will not determine which circuit the inbound packets come from. For example, packets from circuit 1 can be replied to through circuit 1 without any problem, but packets from circuit 2 are also replied to through circuit 1 by default, which may cause not be reachable and time out.

[pfry]

[...]the firewall will not determine which circuit the inbound packets come from[...]

For inbound traffic I would differentiate by interface and destination address; destination alone should be fine. I haven't set it up myself (I avoid hinky routing), but I wouldn't expect it to be a problem (could be a bad assumption). But what advantage would an additional L3 device have?

(Heh: My Internet link died right in the middle of posting this. Trying to sell me on a backup?)