2 WAN Uplinks split routing issues with incoming connections

[paul5012]

Hi,

I've got a problem with a setup as in the drawing:
2 Internet uplinks, each one has a FritzBox. Different providers, each one has a static IPv4 address.
Only FritzBox2 has IPv6 (static address + /56 static prefix, but not of interest here)
OPNsense is 26.4.1-amd64

What I want to have: load balanced WAN links. Services in the DMZ like a Nextcloud or a mail gateway should be reachable via both public (IPv4) adresses.

I followed the intructions in https://docs.opnsense.org/manual/how-tos/multiwan.html and did multiple searches but found no solution.

The gateways table has 3 entries, two for the IPV4 Fritzboxes and one for the IPv6 box. Both v4 gateways have the same priority of 63. I configured a monitor IP (1.1.1.1 and 1.0.0.1 on the other interface).
The gateway is present in the respective WAN interface definition.
There is a gateway group with both v4 gateways, both as tier 1.
Pool options "default", trigger level "packet loss and high latency"

I did not configure DNS servers for each gateway as want unbound to be a full recursor in did not get the point with this part of the story.
I modified the "LAN pass to all" rule as in Step 4.

In the gateway overview one of the Fritzboxes is labeled "active", and there goes all the traffic.
When I try to connect from the internet to the nginx reverse proxy, I succeed when using the address of the "active" Fritzbox.
When I try to access the other public IP the packets are natted from the Fritzbox correctly and the syn packet arrives at the OPNsense. But the syn-ack packet go the wrong interface, with the sender address of the interface where to syn came in.

"Use sticky connections" is on. "shared forwarding" and "Disable force gateway" are off.

What do I miss?

[pfry]

Sounds like you want a policy routing solution (""Gateway" in the rule definitions) for incoming sessions. You'd need to differentiate your policies by destination address. Load-sharing outbound? Got me.

(I'd normally use VRFs for something like this, but hey.)

[paul5012]

would I find something in the documentation, how to achieve this?

[pfry]

Documentation is pretty light... I don't know of any examples. Searching this forum would probably be your best bet for that.

[viragomann]

Is the gateway status shown up as "online" for both IPv4 gateways in System: Gateways: Configuration?

How did you configure firewall rule for incoming traffic?

[wincent]

I thank you need a layer 3 switch that supports PBR to be placed between two FritzBoxes and OPNsense devices.

[paul5012]

yes, both gateways are show as "online".

the firewall rule for incoming traffic is one NAT rule per port, which includes 2 ports (or sometimes even 4, if I want for notebooks or mobile phones to work from the outside (internet) as from the inside (Wifi bridged to LAN) with the same configuration).
I also used the firewall - settings - advanced - Network address translation settings "Reflection for destination NAT" and "Automatic outbound NAT for Reflection".
That seems to work great to get my mobile devices work from in- and outside with only few explicit rules.
But it gives me trouble an another point, and I'm not quite sure how to overrule these settings in an individual rule.
But this is just the next thread.

In the NAT rules I set the "Firewall rule" to "Pass"

[wincent]

This is not a firewall rule issue, it's a routing issue, The default route gateway of the firewall is only one. If the inbound traffic happens to be on your default circuit, there is no problem. If the inbound traffic is on another circuit, the firewall's outbound packets cannot reach this route. You need a layer 3 switch to run PBR to handle this.

[pfry]

[...]You need a layer 3 switch to run PBR to handle this.

? The firewall can handle this under most circumstances.

[wincent]

The firewall only handles the outbound load balancing, for inbound traffic, if PBR is not deployed in your front-end, the firewall will not determine which circuit the inbound packets come from. For example, packets from circuit 1 can be replied to through circuit 1 without any problem, but packets from circuit 2 are also replied to through circuit 1 by default, which may cause not be reachable and time out.

[pfry]

[...]the firewall will not determine which circuit the inbound packets come from[...]

For inbound traffic I would differentiate by interface and destination address; destination alone should be fine. I haven't set it up myself (I avoid hinky routing), but I wouldn't expect it to be a problem (could be a bad assumption). But what advantage would an additional L3 device have?

(Heh: My Internet link died right in the middle of posting this. Trying to sell me on a backup?)

[wincent]

I drew a network topology diagram, and the general structure is like this (the drawing is a bit ugly)
You cannot view this attachment.

[viragomann]

In a multi-WAN setup rerouting the traffic to the correct gateway is normally managend by reply-to.
For this to work it's required that each WAN interface has a gateway assigned, either manually or automatically by DHCP or PPP.
The gateways must be shown up in Interfaces: Overview for the respective WAN.
Futher gateway monitoring must be enabled for each and the status in System: Gateways: Configuration has to be shown up as online.

The reply-to tagging is done by a firewall pass rule, which is defined on one of the WAN interfaces then.
So you have to ensure, that the rule is applied to the incoming (forwarded) traffic. It doesn't work with floating rules or interface group rules, because these could be applied to multiple interfaces. So the reply gateway is not distinct.

Note that group rules and floating rule have precedence over interface rules. So you have to ensure, that there is none of these overriding your WAN rules.
Instead of "pass" in the port forwarding I'd rather create the rules manually on each WAN. State a unique description for each rule, e.g. HTTPS WAN1, HTTP WAN2 and enable logging. So you can check in the log if the WAN2 rule is applied to the incoming traffic.

If it doesn't work either, you can try to explicitly state the WAN2 gateway in the rule at "Reply-to".

However, I'm not certain, if this works a load balancing configuration.
Also there are several threads here, complaining about trouble with reply-to in 26.1.

An L3 switch in front of OPNsense can only help here, if it does hairpinning incoming traffic to its interface IP. But this removes the information about origin source IP and is probably unwanted.

[paul5012]

The firewall only handles the outbound load balancing, for inbound traffic, if PBR is not deployed in your front-end, the firewall will not determine which circuit the inbound packets come from. For example, packets from circuit 1 can be replied to through circuit 1 without any problem, but packets from circuit 2 are also replied to through circuit 1 by default, which may cause not be reachable and time out.

That is exactly the problem.

And I cannot imagine, that OPNsense as kind of mature firewall is not able to solve this.
Allthough it startles me more and more that I find nothing to this setup, even not in the book that comes with the business edition.

Should I regret my decision towards OPNsense as successor of the today going out-of-service Sophos UTM?

[wincent]

I think HAProxy is an option for load balancing.