2 WAN Uplinks split routing issues with incoming connections

[paul5012]

the problem hits every incoming connection: Nextcloud web traffic, incoming mail traffic (which is not handled by the OPNsense; I offloaded this to a Proxmox Mail Gateway, to reduce complexity of the OPNsense setup), incoming VPN connections

@viragomann: you suggest tagging of the incoming connections, and to further routing based on this tagging?
that sounds promising to me. Allthough I not yet did this. Have to dig into this.

[viragomann]

Tagging of incoming traffic should be done automatically by the firewall rule on the WAN interface, which passes it. You should just get sure, that the respective interface pass rule is applied to the traffic, but no other (floating or group).
OPNsense should route the replies accordingly to the reply-to tags.

If no success either, you can state the reply-to gateway in each rule manually.
But anyway you have to ensure that the respective rule is applied. This presumes that you state a unique name for the rule and enable logging.

[paul5012]

slowly I see progress.
I was not aware that "Rules [new]" section is possibly not (yet) stuffed with all features of the old system.
So far I had tried to live with destination NAT rules and the option "Firewall rule": "Register rule".

(When talking about this dialog, i.e. it's advanced options:
 1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?
 2) "NAT reflection" I had tried this in a former approach which ended in a complete disaster, eventually. I freshly installed the OPNsense from scratch as it seemed I could never more reverse settings. In the current approach I omitted all these possibly in other scenarios helpful setting.
 3) "Set tag" I suspected to be what I should use. Read on to my current solution.
 )

So I dropped the idea with "Firewall rule" "Register rule" or "pass" as it gave not the requested solution, went back to "Manual"

Next approach: define a "Rules [new]" rule. To make it short: using all the possible options in this specific dialogue did not help me.
The solution (currently I hope it to be the solution) is in the old "Rules - [incoming interface]" dialogue:
Only there, in the "Advanced features" section, is the entry "reply-to". Where I can choose the gateway of this specific interface.
Now the packets flow as expected, I can reach my services with both published IP addresses.
So I have to define such a rule for every port forwarding for each of the two upstream interfaces?

I'd expect this to be default behaviour, that return packets go back the interface they came in.

Do I get something wrong?

[Patrick M. Hausen]

The default behaviour of any standard Unix like system without policy based routing is to always follow the routing table for egress no matter which way a matching ingress packet arrived.

The default in OPNsense has always been to override that behaviour for WAN interfaces: "reply-to" is enabled globally for interfaces that have an explicit gateway set.

You can disable that feature at Firewall > Settings > Advanced > Disable reply-to. Maybe that is the case?

[viragomann]

I was not aware that "Rules [new]" section is possibly not (yet) stuffed with all features of the old system.

It is already, as far as I know.

However, it's not recommended to use both, old and new rules.
You should migrate your rule at some point.

So far I had tried to live with destination NAT rules and the option "Firewall rule": "Register rule".

With "Register rule" OPNsense creates the firewall rule for you and you're able to modify it later. But I'd rather create rule manually instead.

1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?

For a single port forwarding rule, it does.
I guess, this can make sense if you forward multiple / all ports. So using this option you can define an exception.

2) "NAT reflection"

NAT reflection mirrors a NAT rule to internal interface in short.
So if you define a NAT rule on WAN for the WAN IP to forward port 443 to webserver 1 in DMZ. NAT reflection enable you to access the webserver using the WAN IP also from LAN.

3) "Set tag" I suspected to be what I should use.

No, this is for custom tagging.
With this you can instruct OPNsense to tag connection and use this tag by following rule, e.g. outbound rules.

Now the packets flow as expected, I can reach my services with both published IP addresses.

Fine.

So I have to define such a rule for every port forwarding for each of the two upstream interfaces?

You only need this for incoming traffic from the internet (from source addresses, which OPNsense has no specific route for. This could also be a remote client accessing your resources over VPN).
You can also use aliases for forwarded ports or destination IPs. So don't need one for each NAT rule.

I'd expect this to be default behaviour, that return packets go back the interface they came in.

Yes, it is. But possibly this doesn't work together with a load balancing gateway group.

[paul5012]

The default behaviour of any standard Unix like system without policy based routing is to always follow the routing table for egress no matter which way a matching ingress packet arrived.

The default in OPNsense has always been to override that behaviour for WAN interfaces: "reply-to" is enabled globally for interfaces that have an explicit gateway set.

ack

You can disable that feature at Firewall > Settings > Advanced > Disable reply-to. Maybe that is the case?

no

[paul5012]

I was not aware that "Rules [new]" section is possibly not (yet) stuffed with all features of the old system.

It is already, as far as I know.

However, it's not recommended to use both, old and new rules.
You should migrate your rule at some point.

I had done this.

So far I had tried to live with destination NAT rules and the option "Firewall rule": "Register rule".

With "Register rule" OPNsense creates the firewall rule for you and you're able to modify it later. But I'd rather create rule manually instead.

what I did, finally.

1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?

For a single port forwarding rule, it does.
I guess, this can make sense if you forward multiple / all ports. So using this option you can define an exception.

Still not quite clear to me.

2) "NAT reflection"

NAT reflection mirrors a NAT rule to internal interface in short.
So if you define a NAT rule on WAN for the WAN IP to forward port 443 to webserver 1 in DMZ. NAT reflection enable you to access the webserver using the WAN IP also from LAN.

got this. seemed to help me alot, but gave me other troubles when a forward from port 25 to [dmz-server] port 25 suddenly hat a redirection from the DMz system to itself.
Could not get rid of this afterwards, what led to the fresh install eventually.

3) "Set tag" I suspected to be what I should use.

No, this is for custom tagging.
With this you can instruct OPNsense to tag connection and use this tag by following rule, e.g. outbound rules.

Now the packets flow as expected, I can reach my services with both published IP addresses.

Fine.

So I have to define such a rule for every port forwarding for each of the two upstream interfaces?

You only need this for incoming traffic from the internet (from source addresses, which OPNsense has no specific route for. This could also be a remote client accessing your resources over VPN).
You can also use aliases for forwarded ports or destination IPs. So don't need one for each NAT rule.

I use these aliases excessively.

I'd expect this to be default behaviour, that return packets go back the interface they came in.

Yes, it is. But possibly this doesn't work together with a load balancing gateway group.

I do (not currently) make use of any loadbalancing gateway group.

When you said, "new rules" should be able to replace the old style completely: I could not find a way to define the back path as needed.
Could it be possible that there is a bug or a to-be-implemented feature somewhere around in this version?

[lmoore]

1) what is the sense of "no RDR (NOT)" (Enabling this option will disable redirection for traffic matching this rule.) Wouldn't make this the rule itself pointless?

For a single port forwarding rule, it does.
I guess, this can make sense if you forward multiple / all ports. So using this option you can define an exception.

Still not quite clear to me.

OPNsense uses "no rdr" by default on the LAN interface to prevent redirects away from OPNsense management ports, i.e. SSH, HTTP & HTTPS and for CARP.



HTH.