Hi,
I've got a problem with a setup as in the drawing:
2 Internet uplinks, each one has a FritzBox. Different providers, each one has a static IPv4 address.
Only FritzBox2 has IPv6 (static address + /56 static prefix, but not of interest here)
OPNsense is 26.4.1-amd64
What I want to have: load balanced WAN links. Services in the DMZ like a Nextcloud or a mail gateway should be reachable via both public (IPv4) adresses.
I followed the intructions in https://docs.opnsense.org/manual/how-tos/multiwan.html and did multiple searches but found no solution.
The gateways table has 3 entries, two for the IPV4 Fritzboxes and one for the IPv6 box. Both v4 gateways have the same priority of 63. I configured a monitor IP (1.1.1.1 and 1.0.0.1 on the other interface).
The gateway is present in the respective WAN interface definition.
There is a gateway group with both v4 gateways, both as tier 1.
Pool options "default", trigger level "packet loss and high latency"
I did not configure DNS servers for each gateway as want unbound to be a full recursor in did not get the point with this part of the story.
I modified the "LAN pass to all" rule as in Step 4.
In the gateway overview one of the Fritzboxes is labeled "active", and there goes all the traffic.
When I try to connect from the internet to the nginx reverse proxy, I succeed when using the address of the "active" Fritzbox.
When I try to access the other public IP the packets are natted from the Fritzbox correctly and the syn packet arrives at the OPNsense. But the syn-ack packet go the wrong interface, with the sender address of the interface where to syn came in.
"Use sticky connections" is on. "shared forwarding" and "Disable force gateway" are off.
What do I miss?
Sounds like you want a policy routing solution (""Gateway" in the rule definitions) for incoming sessions. You'd need to differentiate your policies by destination address. Load-sharing outbound? Got me.
(I'd normally use VRFs for something like this, but hey.)
would I find something in the documentation, how to achieve this?
Documentation (https://docs.opnsense.org/manual/firewall.html#id3) is pretty light... I don't know of any examples. Searching this forum would probably be your best bet for that.
Is the gateway status shown up as "online" for both IPv4 gateways in System: Gateways: Configuration?
How did you configure firewall rule for incoming traffic?
I thank you need a layer 3 switch that supports PBR to be placed between two FritzBoxes and OPNsense devices.
yes, both gateways are show as "online".
the firewall rule for incoming traffic is one NAT rule per port, which includes 2 ports (or sometimes even 4, if I want for notebooks or mobile phones to work from the outside (internet) as from the inside (Wifi bridged to LAN) with the same configuration).
I also used the firewall - settings - advanced - Network address translation settings "Reflection for destination NAT" and "Automatic outbound NAT for Reflection".
That seems to work great to get my mobile devices work from in- and outside with only few explicit rules.
But it gives me trouble an another point, and I'm not quite sure how to overrule these settings in an individual rule.
But this is just the next thread.
In the NAT rules I set the "Firewall rule" to "Pass"
This is not a firewall rule issue, it's a routing issue, The default route gateway of the firewall is only one. If the inbound traffic happens to be on your default circuit, there is no problem. If the inbound traffic is on another circuit, the firewall's outbound packets cannot reach this route. You need a layer 3 switch to run PBR to handle this.
Quote from: wincent on June 29, 2026, 11:04:25 AM[...]You need a layer 3 switch to run PBR to handle this.
? The firewall can handle this under most circumstances.
The firewall only handles the outbound load balancing, for inbound traffic, if PBR is not deployed in your front-end, the firewall will not determine which circuit the inbound packets come from. For example, packets from circuit 1 can be replied to through circuit 1 without any problem, but packets from circuit 2 are also replied to through circuit 1 by default, which may cause not be reachable and time out.
Quote from: wincent on June 30, 2026, 04:01:15 AM[...]the firewall will not determine which circuit the inbound packets come from[...]
For inbound traffic I would differentiate by interface and destination address; destination alone should be fine. I haven't set it up myself (I avoid hinky routing), but I wouldn't expect it to be a problem (could be a bad assumption). But what advantage would an additional L3 device have?
(Heh: My Internet link died right in the middle of posting this. Trying to sell me on a backup?)
I drew a network topology diagram, and the general structure is like this (the drawing is a bit ugly)
PBR.png
In a multi-WAN setup rerouting the traffic to the correct gateway is normally managend by reply-to.
For this to work it's required that each WAN interface has a gateway assigned, either manually or automatically by DHCP or PPP.
The gateways must be shown up in Interfaces: Overview for the respective WAN.
Futher gateway monitoring must be enabled for each and the status in System: Gateways: Configuration has to be shown up as online.
The reply-to tagging is done by a firewall pass rule, which is defined on one of the WAN interfaces then.
So you have to ensure, that the rule is applied to the incoming (forwarded) traffic. It doesn't work with floating rules or interface group rules, because these could be applied to multiple interfaces. So the reply gateway is not distinct.
Note that group rules and floating rule have precedence over interface rules. So you have to ensure, that there is none of these overriding your WAN rules.
Instead of "pass" in the port forwarding I'd rather create the rules manually on each WAN. State a unique description for each rule, e.g. HTTPS WAN1, HTTP WAN2 and enable logging. So you can check in the log if the WAN2 rule is applied to the incoming traffic.
If it doesn't work either, you can try to explicitly state the WAN2 gateway in the rule at "Reply-to".
However, I'm not certain, if this works a load balancing configuration.
Also there are several threads here, complaining about trouble with reply-to in 26.1.
An L3 switch in front of OPNsense can only help here, if it does hairpinning incoming traffic to its interface IP. But this removes the information about origin source IP and is probably unwanted.
Quote from: wincent on June 30, 2026, 04:01:15 AMThe firewall only handles the outbound load balancing, for inbound traffic, if PBR is not deployed in your front-end, the firewall will not determine which circuit the inbound packets come from. For example, packets from circuit 1 can be replied to through circuit 1 without any problem, but packets from circuit 2 are also replied to through circuit 1 by default, which may cause not be reachable and time out.
That is exactly the problem.
And I cannot imagine, that OPNsense as kind of mature firewall is not able to solve this.
Allthough it startles me more and more that I find nothing to this setup, even not in the book that comes with the business edition.
Should I regret my decision towards OPNsense as successor of the today going out-of-service Sophos UTM?
I think HAProxy is an option for load balancing.
the problem hits every incoming connection: Nextcloud web traffic, incoming mail traffic (which is not handled by the OPNsense; I offloaded this to a Proxmox Mail Gateway, to reduce complexity of the OPNsense setup), incoming VPN connections
@viragomann: you suggest tagging of the incoming connections, and to further routing based on this tagging?
that sounds promising to me. Allthough I not yet did this. Have to dig into this.
Tagging of incoming traffic should be done automatically by the firewall rule on the WAN interface, which passes it. You should just get sure, that the respective interface pass rule is applied to the traffic, but no other (floating or group).
OPNsense should route the replies accordingly to the reply-to tags.
If no success either, you can state the reply-to gateway in each rule manually.
But anyway you have to ensure that the respective rule is applied. This presumes that you state a unique name for the rule and enable logging.
slowly I see progress.
I was not aware that "Rules [new]" section is possibly not (yet) stuffed with all features of the old system.
So far I had tried to live with destination NAT rules and the option "Firewall rule": "Register rule".
(When talking about this dialog, i.e. it's advanced options:
1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?
2) "NAT reflection" I had tried this in a former approach which ended in a complete disaster, eventually. I freshly installed the OPNsense from scratch as it seemed I could never more reverse settings. In the current approach I omitted all these possibly in other scenarios helpful setting.
3) "Set tag" I suspected to be what I should use. Read on to my current solution.
)
So I dropped the idea with "Firewall rule" "Register rule" or "pass" as it gave not the requested solution, went back to "Manual"
Next approach: define a "Rules [new]" rule. To make it short: using all the possible options in this specific dialogue did not help me.
The solution (currently I hope it to be the solution) is in the old "Rules - [incoming interface]" dialogue:
Only there, in the "Advanced features" section, is the entry "reply-to". Where I can choose the gateway of this specific interface.
Now the packets flow as expected, I can reach my services with both published IP addresses.
So I have to define such a rule for every port forwarding for each of the two upstream interfaces?
I'd expect this to be default behaviour, that return packets go back the interface they came in.
Do I get something wrong?
The default behaviour of any standard Unix like system without policy based routing is to always follow the routing table for egress no matter which way a matching ingress packet arrived.
The default in OPNsense has always been to override that behaviour for WAN interfaces: "reply-to" is enabled globally for interfaces that have an explicit gateway set.
You can disable that feature at Firewall > Settings > Advanced > Disable reply-to. Maybe that is the case?
Quote from: paul5012 on June 30, 2026, 03:38:23 PMI was not aware that "Rules [new]" section is possibly not (yet) stuffed with all features of the old system.
It is already, as far as I know.
However, it's not recommended to use both, old and new rules.
You should migrate your rule at some point.
Quote from: paul5012 on June 30, 2026, 03:38:23 PMSo far I had tried to live with destination NAT rules and the option "Firewall rule": "Register rule".
With "Register rule" OPNsense creates the firewall rule for you and you're able to modify it later. But I'd rather create rule manually instead.
Quote from: paul5012 on June 30, 2026, 03:38:23 PM1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?
For a single port forwarding rule, it does.
I guess, this can make sense if you forward multiple / all ports. So using this option you can define an exception.
Quote from: paul5012 on June 30, 2026, 03:38:23 PM2) "NAT reflection"
NAT reflection mirrors a NAT rule to internal interface in short.
So if you define a NAT rule on WAN for the WAN IP to forward port 443 to webserver 1 in DMZ. NAT reflection enable you to access the webserver using the WAN IP also from LAN.
Quote from: paul5012 on June 30, 2026, 03:38:23 PM3) "Set tag" I suspected to be what I should use.
No, this is for custom tagging.
With this you can instruct OPNsense to tag connection and use this tag by following rule, e.g. outbound rules.
Quote from: paul5012 on June 30, 2026, 03:38:23 PMNow the packets flow as expected, I can reach my services with both published IP addresses.
Fine.
Quote from: paul5012 on June 30, 2026, 03:38:23 PMSo I have to define such a rule for every port forwarding for each of the two upstream interfaces?
You only need this for incoming traffic from the internet (from source addresses, which OPNsense has no specific route for. This could also be a remote client accessing your resources over VPN).
You can also use aliases for forwarded ports or destination IPs. So don't need one for each NAT rule.
Quote from: paul5012 on June 30, 2026, 03:38:23 PMI'd expect this to be default behaviour, that return packets go back the interface they came in.
Yes, it is. But possibly this doesn't work together with a load balancing gateway group.
Quote from: Patrick M. Hausen on June 30, 2026, 04:14:38 PMThe default behaviour of any standard Unix like system without policy based routing is to always follow the routing table for egress no matter which way a matching ingress packet arrived.
The default in OPNsense has always been to override that behaviour for WAN interfaces: "reply-to" is enabled globally for interfaces that have an explicit gateway set.
ack
Quote from: Patrick M. Hausen on June 30, 2026, 04:14:38 PMYou can disable that feature at Firewall > Settings > Advanced > Disable reply-to. Maybe that is the case?
no
Quote from: viragomann on June 30, 2026, 04:21:15 PMQuote from: paul5012 on June 30, 2026, 03:38:23 PMI was not aware that "Rules [new]" section is possibly not (yet) stuffed with all features of the old system.
It is already, as far as I know.
However, it's not recommended to use both, old and new rules.
You should migrate your rule at some point.
I had done this.
Quote from: viragomann on June 30, 2026, 04:21:15 PMQuote from: paul5012 on June 30, 2026, 03:38:23 PMSo far I had tried to live with destination NAT rules and the option "Firewall rule": "Register rule".
With "Register rule" OPNsense creates the firewall rule for you and you're able to modify it later. But I'd rather create rule manually instead.
what I did, finally.
Quote from: viragomann on June 30, 2026, 04:21:15 PMQuote from: paul5012 on June 30, 2026, 03:38:23 PM1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?
For a single port forwarding rule, it does.
I guess, this can make sense if you forward multiple / all ports. So using this option you can define an exception.
Still not quite clear to me.
Quote from: viragomann on June 30, 2026, 04:21:15 PMQuote from: paul5012 on June 30, 2026, 03:38:23 PM2) "NAT reflection"
NAT reflection mirrors a NAT rule to internal interface in short.
So if you define a NAT rule on WAN for the WAN IP to forward port 443 to webserver 1 in DMZ. NAT reflection enable you to access the webserver using the WAN IP also from LAN.
got this. seemed to help me alot, but gave me other troubles when a forward from port 25 to [dmz-server] port 25 suddenly hat a redirection from the DMz system to itself.
Could not get rid of this afterwards, what led to the fresh install eventually.
Quote from: viragomann on June 30, 2026, 04:21:15 PMQuote from: paul5012 on June 30, 2026, 03:38:23 PM3) "Set tag" I suspected to be what I should use.
No, this is for custom tagging.
With this you can instruct OPNsense to tag connection and use this tag by following rule, e.g. outbound rules.
Quote from: paul5012 on June 30, 2026, 03:38:23 PMNow the packets flow as expected, I can reach my services with both published IP addresses.
Fine.
Quote from: paul5012 on June 30, 2026, 03:38:23 PMSo I have to define such a rule for every port forwarding for each of the two upstream interfaces?
You only need this for incoming traffic from the internet (from source addresses, which OPNsense has no specific route for. This could also be a remote client accessing your resources over VPN).
You can also use aliases for forwarded ports or destination IPs. So don't need one for each NAT rule.
I use these aliases excessively.
Quote from: viragomann on June 30, 2026, 04:21:15 PMQuote from: paul5012 on June 30, 2026, 03:38:23 PMI'd expect this to be default behaviour, that return packets go back the interface they came in.
Yes, it is. But possibly this doesn't work together with a load balancing gateway group.
I do (not currently) make use of any loadbalancing gateway group.
When you said, "new rules" should be able to replace the old style completely: I could not find a way to define the back path as needed.
Could it be possible that there is a bug or a to-be-implemented feature somewhere around in this version?
Quote from: paul5012 on June 30, 2026, 09:57:00 PMQuoteQuote1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?
For a single port forwarding rule, it does.
I guess, this can make sense if you forward multiple / all ports. So using this option you can define an exception.
Still not quite clear to me.
OPNsense uses "no rdr" by default on the LAN interface to prevent redirects away from OPNsense management ports, i.e. SSH, HTTP & HTTPS and for CARP.
(https://forum.opnsense.org/index.php?action=dlattach;attach=56245)
HTH.
Quote from: lmoore on July 01, 2026, 02:51:24 AMQuote from: paul5012 on June 30, 2026, 09:57:00 PMQuoteQuote1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?
For a single port forwarding rule, it does.
I guess, this can make sense if you forward multiple / all ports. So using this option you can define an exception.
Still not quite clear to me.
OPNsense uses "no rdr" by default on the LAN interface to prevent redirects away from OPNsense management ports, i.e. SSH, HTTP & HTTPS and for CARP.
HTH.
I see. So it should better read "no auto RDR rules"?
I mean this option exists in a destination NAT dialogue. Which is a redirect, isn't it? So I still find it confusing to have an option "no rdr". Whilst with that auto-RDR mechanisms this gets an other view. And if then "no auto RDR rules" can be choosen that makes sense.
Or do I still miss the meaning of the terms and mechanisms?
Summary: problem solved
allthough it was excepted (as some contributors wrote here, thanks for the replies) that an automatic reply-to setting should make the
response packets leave the firewall towards the proper gateway I could not see this happen. Always the default route was choosen.
So
1) I do not (as I did in the beginning) choose some kind of automatic firewall generation in the DNAT rules, but choose "manual" rule creation
2) these manual rules can be set with the "Rules [new]" dialogue, but need "Advanced mode".
there I had to configure under "Source Routing" the "Reply-to" to point to the correct uplink gateway.
This means that I have one such rule per target host, target port and incoming interface.
Technically one could combine multiple target hosts and ports into one rule but that would be less specific then and allow additional traffic possibly.
special thanks to @viragomann