Messages

Summary: problem solved
allthough it was excepted (as some contributors wrote here, thanks for the replies) that an automatic reply-to setting should make the
response packets leave the firewall towards the proper gateway I could not see this happen. Always the default route was choosen.
So
1) I do not (as I did in the beginning) choose some kind of automatic firewall generation in the DNAT rules, but choose "manual" rule creation
2) these manual rules can be set with the "Rules [new]" dialogue, but need "Advanced mode".
   there I had to configure under "Source Routing" the "Reply-to" to point to the correct uplink gateway.
   This means that I have one such rule per target host, target port and incoming interface.
   Technically one could combine multiple target hosts and ports into one rule but that would be less specific then and allow additional traffic possibly.

special thanks to @viragomann
 

1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?

For a single port forwarding rule, it does.
I guess, this can make sense if you forward multiple / all ports. So using this option you can define an exception.

Still not quite clear to me.

OPNsense uses "no rdr" by default on the LAN interface to prevent redirects away from OPNsense management ports, i.e. SSH, HTTP & HTTPS and for CARP.

HTH.

I see. So it should better read "no auto RDR rules"?

I mean this option exists in a destination NAT dialogue. Which is a redirect, isn't it? So I still find it confusing to have an option "no rdr". Whilst with that auto-RDR mechanisms this gets an other view. And if then "no auto RDR rules" can be choosen that makes sense.

Or do I still miss the meaning of the terms and mechanisms?

I was not aware that "Rules [new]" section is possibly not (yet) stuffed with all features of the old system.

It is already, as far as I know.

However, it's not recommended to use both, old and new rules.
You should migrate your rule at some point.

I had done this.

So far I had tried to live with destination NAT rules and the option "Firewall rule": "Register rule".

With "Register rule" OPNsense creates the firewall rule for you and you're able to modify it later. But I'd rather create rule manually instead.

what I did, finally.

1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?

For a single port forwarding rule, it does.
I guess, this can make sense if you forward multiple / all ports. So using this option you can define an exception.

Still not quite clear to me.

2) "NAT reflection"

NAT reflection mirrors a NAT rule to internal interface in short.
So if you define a NAT rule on WAN for the WAN IP to forward port 443 to webserver 1 in DMZ. NAT reflection enable you to access the webserver using the WAN IP also from LAN.

got this. seemed to help me alot, but gave me other troubles when a forward from port 25 to [dmz-server] port 25 suddenly hat a redirection from the DMz system to itself.
Could not get rid of this afterwards, what led to the fresh install eventually.

3) "Set tag" I suspected to be what I should use.

No, this is for custom tagging.
With this you can instruct OPNsense to tag connection and use this tag by following rule, e.g. outbound rules.

Now the packets flow as expected, I can reach my services with both published IP addresses.

Fine.

So I have to define such a rule for every port forwarding for each of the two upstream interfaces?

You only need this for incoming traffic from the internet (from source addresses, which OPNsense has no specific route for. This could also be a remote client accessing your resources over VPN).
You can also use aliases for forwarded ports or destination IPs. So don't need one for each NAT rule.

I use these aliases excessively.

I'd expect this to be default behaviour, that return packets go back the interface they came in.

Yes, it is. But possibly this doesn't work together with a load balancing gateway group.

I do (not currently) make use of any loadbalancing gateway group.

When you said, "new rules" should be able to replace the old style completely: I could not find a way to define the back path as needed.
Could it be possible that there is a bug or a to-be-implemented feature somewhere around in this version?

The default behaviour of any standard Unix like system without policy based routing is to always follow the routing table for egress no matter which way a matching ingress packet arrived.

The default in OPNsense has always been to override that behaviour for WAN interfaces: "reply-to" is enabled globally for interfaces that have an explicit gateway set.

ack

You can disable that feature at Firewall > Settings > Advanced > Disable reply-to. Maybe that is the case?

no

slowly I see progress.
I was not aware that "Rules [new]" section is possibly not (yet) stuffed with all features of the old system.
So far I had tried to live with destination NAT rules and the option "Firewall rule": "Register rule".

(When talking about this dialog, i.e. it's advanced options:
 1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?
 2) "NAT reflection" I had tried this in a former approach which ended in a complete disaster, eventually. I freshly installed the OPNsense from scratch as it seemed I could never more reverse settings. In the current approach I omitted all these possibly in other scenarios helpful setting.
 3) "Set tag" I suspected to be what I should use. Read on to my current solution.
 )

So I dropped the idea with "Firewall rule" "Register rule" or "pass" as it gave not the requested solution, went back to "Manual"

Next approach: define a "Rules [new]" rule. To make it short: using all the possible options in this specific dialogue did not help me.
The solution (currently I hope it to be the solution) is in the old "Rules - [incoming interface]" dialogue:
Only there, in the "Advanced features" section, is the entry "reply-to". Where I can choose the gateway of this specific interface.
Now the packets flow as expected, I can reach my services with both published IP addresses.
So I have to define such a rule for every port forwarding for each of the two upstream interfaces?

I'd expect this to be default behaviour, that return packets go back the interface they came in.

Do I get something wrong?

the problem hits every incoming connection: Nextcloud web traffic, incoming mail traffic (which is not handled by the OPNsense; I offloaded this to a Proxmox Mail Gateway, to reduce complexity of the OPNsense setup), incoming VPN connections

@viragomann: you suggest tagging of the incoming connections, and to further routing based on this tagging?
that sounds promising to me. Allthough I not yet did this. Have to dig into this.

The firewall only handles the outbound load balancing, for inbound traffic, if PBR is not deployed in your front-end, the firewall will not determine which circuit the inbound packets come from. For example, packets from circuit 1 can be replied to through circuit 1 without any problem, but packets from circuit 2 are also replied to through circuit 1 by default, which may cause not be reachable and time out.

That is exactly the problem.

And I cannot imagine, that OPNsense as kind of mature firewall is not able to solve this.
Allthough it startles me more and more that I find nothing to this setup, even not in the book that comes with the business edition.

Should I regret my decision towards OPNsense as successor of the today going out-of-service Sophos UTM?

yes, both gateways are show as "online".

the firewall rule for incoming traffic is one NAT rule per port, which includes 2 ports (or sometimes even 4, if I want for notebooks or mobile phones to work from the outside (internet) as from the inside (Wifi bridged to LAN) with the same configuration).
I also used the firewall - settings - advanced - Network address translation settings "Reflection for destination NAT" and "Automatic outbound NAT for Reflection".
That seems to work great to get my mobile devices work from in- and outside with only few explicit rules.
But it gives me trouble an another point, and I'm not quite sure how to overrule these settings in an individual rule.
But this is just the next thread.

In the NAT rules I set the "Firewall rule" to "Pass"

would I find something in the documentation, how to achieve this?

Hi,

I've got a problem with a setup as in the drawing:
2 Internet uplinks, each one has a FritzBox. Different providers, each one has a static IPv4 address.
Only FritzBox2 has IPv6 (static address + /56 static prefix, but not of interest here)
OPNsense is 26.4.1-amd64

What I want to have: load balanced WAN links. Services in the DMZ like a Nextcloud or a mail gateway should be reachable via both public (IPv4) adresses.

I followed the intructions in https://docs.opnsense.org/manual/how-tos/multiwan.html and did multiple searches but found no solution.

The gateways table has 3 entries, two for the IPV4 Fritzboxes and one for the IPv6 box. Both v4 gateways have the same priority of 63. I configured a monitor IP (1.1.1.1 and 1.0.0.1 on the other interface).
The gateway is present in the respective WAN interface definition.
There is a gateway group with both v4 gateways, both as tier 1.
Pool options "default", trigger level "packet loss and high latency"

I did not configure DNS servers for each gateway as want unbound to be a full recursor in did not get the point with this part of the story.
I modified the "LAN pass to all" rule as in Step 4.

In the gateway overview one of the Fritzboxes is labeled "active", and there goes all the traffic.
When I try to connect from the internet to the nginx reverse proxy, I succeed when using the address of the "active" Fritzbox.
When I try to access the other public IP the packets are natted from the Fritzbox correctly and the syn packet arrives at the OPNsense. But the syn-ack packet go the wrong interface, with the sender address of the interface where to syn came in.

"Use sticky connections" is on. "shared forwarding" and "Disable force gateway" are off.

What do I miss?