I was recently testing a few things related to a thread on the forum and stumbled upon this behavior that I can't explain and fix.
My test setup is as follows:
ONT - OuterOPN - Switch - PC1 in VLAN1
- PC2 in VLAN2
- InnerOPN with WAN in VLAN2 - NUC running Ubuntu
Both OPNs are running 24.7.12 virtualized under proxmox. WAN and LAN map to proxmox bridges currently only used by OPN.
InnerOPN has WAN in a VLAN constrained to Internet only so my experiments can't interfere with my internal network.
VLAN2 is a /29 network with DHCP and 2 "hosts", PC2 and InnerOPN WAN.
All switch ports (apart from the uplink trunk) are access ports. IOW, I didn't configure VLANs on the InnerOPN. Traffic is tagged/untagged at the switch.
InnerOPN was just reset to default with the following configuration changes:
* Disabled 80->443 redirection to free port 80 (likely not strictly necessary)
* Disabled bogons/privates on WAN
* Port forward of HTTP to the NUC (running nginx) with associated rule.
I can ping PC2 from the NUC, as expected.
I can access http://InnerOPN.WAN_IP from PC1 and I get the welcome to nginx page.
But doing the same thing from PC2 results in timeouts.
I observe the following things:
* Proper redirection at InnerOPN to NUC, blue/green entries in the FW live view
* State violation on OuterOPN interface VLAN2 corresponding to reply traffic from InnerOPN WAN (expected failure given the absence of state)
* Doing packet captures, the HTTP replies out of InnerOPN WAN feature the MAC address of OuterOPN LAN (i.e. the gateway of VLAN2)
* In my initial experiment (I reran it today), PC2 was present in the ARP table with the correct address.
I don't understand what's going on.
Outbound traffic is going straight to PC2 as expected (ICMP use case).
Reply traffic in the same direction is sent to the GW (HTTP use case).
The fact that the test succeeds from PC1 can be explained by the fact that OuterOPN processes the request (inter-VLAN) so it's appropriate for the reply to go through the GW as well.
Patrick had mentioned that this was expected in a lab setting and suggested checking "disable force gateway".
Beyond the fact that I don't understand fully the FW rule behind this setting, I don't believe it applies and in fact did nothing in my initial experiment.
I understand that this may not be a concern for the general use case. Communication with a neighbor of the WAN side of OPN is likely rare.
But there must be a reason for this behavior, and maybe a setting to override it.
Only skim-read this, but sounds like the reply-to problem. Does [Firewall -> Settings -> Advanced -> Disable reply-to] fix it?
Quote from: dseven on January 20, 2025, 11:22:30 AMOnly skim-read this, but sounds like the reply-to problem. Does [Firewall -> Settings -> Advanced -> Disable reply-to] fix it?
It indeed does. Thanks!
I had seen this setting, but the inline explanation is less than crystal clear to me (multi-WAN, bridging... none of them applied).
I should have looked at the doc, which has this warning:
QuoteAlthough our default is to enable this rule for historic reasons, there are side-affects when adding reply-to to every "wan" type rule. When allowing traffic originating from the same network as the interface is attached to, it will still reply the packet to the configured gateway.
And now that I know the name of the setting, I also found this here (https://docs.opnsense.org/manual/firewall.html#policy-based-routing)
Quotereply-to
By default traffic is always send to the connected gateway on the interface. If for some reason you don't want to force traffic to that gateway, you can disable this behaviour or enforce an alternative target here.
This is not totally over though. The second part of the warning here (https://docs.opnsense.org/manual/firewall_settings.html#disable-reply-to) implies there's a way to influence reply-to on a per rule basis.
There's also "or enforce an alternative target here" in that last quote.
So my question left is: where is reply-to exposed in the rules?
IOW, can I see the effect of the "disable reply-to" in any rule? My rules seem unchanged.
Quote from: EricPerl on January 20, 2025, 09:32:41 PMSo my question left is: where is reply-to exposed in the rules?
Show "Advanced features" for a rule, then look for "reply-to".
QuoteIOW, can I see the effect of the "disable reply-to" in any rule? My rules seem unchanged.
When reply-to is in effect (i.e. not disabled), you should see it on the pf rules for your WAN interface(s) - `pfctl -s rules | grep reply-to` in a shell....
Thanks again!
I indeed see it in the GUI on my Wireguard rule on OuterOPN. I don't remember ever opening that "Advanced Features" section. Wow...
And since the PF associated rule is only present in the list (can't open it, nor the autogenerated rules), it was completely undiscoverable on InnerOPN.
The CLI command shows something after re-enabling the setting on InnerOPN.