How is my pass rule different from default (which works, while mine doesn't)?

[eth0]

Another newbie question...

I'm trying to access a bridged cable modem's web UI that responds at 192.168.100.1:80 on the WAN interface. This works fine using the default Floating rule "let out anything from firewall host itself" (pass IPv4+6 to any).

However if I define my own WAN firewall rule (pass IPv4 to 192.168.100.1 only), so I could also block any other private addresses from leaking out on WAN, the browser can no longer load the UI (server not responding). There's nothing else blocking it, I have turned on logging on all WAN and Floating rules.

I can't view the actual definition of the default rule beyond what's shown in the firewall rules list, but on that level I don't see any difference to my own rule.

Could this be some kind of a gateway issue, as the modem's IP is outside of the WAN interface's subnet?

Doing a packet capture on WAN with the default rule, I see that communication happens between the firewall host's WAN address and 192.168.100.1, and the session soon proceeds into HTTP headers and such. With my own rule, after each response from 192.168.100.1 to the firewall host, it sends (or tries) another response to 192.168.0.101 (the laptop where I'm running the browser), and this repeats until the browser times out. I don't know why this is happening.

I think at some point I was able to make the default rule fail by changing some gateway related setting, but can't recall what it was and can't find it again. (I could be wrong and it may have failed for some unrelated reason.)

I've also tried changing the Gateway setting in my own rule but it won't let me (Policy based routing is only supported on inbound rules).

Is there some special magic going on in the default rule that I'm missing?

[chemlud]

Rules on WAN are evaluated for traffic starting from WAN. Your rule would need to be on the respective LAN interface to allow anything outgoing to another interface (including WAN).

[eth0]

Doesn't that depend on whether it's defined as an "in" rule or an "out" rule?

Both the explicit rule that I tried to create, and the default floating rule that gets applied if I remove my own rule, are evaluated on WAN (as "out" rules). I can see this in the log, and I can see in packet capture that actual packets are sent and received.

Certainly you also need to have an "in" rule on the LAN to allow packets into the firewall on the browser end of things, but this is taken care of in the default rules (allow LAN to any), otherwise there wouldn't be anything to match or capture on the WAN side I think.

Or am I missing something really fundamental here?

[eth0]

I did some more digging in the capture log. I was wrong before, it's not the cable modem trying to send packets to a LAN address directly. What seems to be happening is that OPNsense echoes packets from the cable modem back to WAN, with a LAN destination address.

To understand this I tried to make a table out of the relevant data. I'm not sure that I got everything right, but I think the following is roughly what happens. In the tables below "modem" and "WLAN" represent MAC addresses, and when not noted, the connection is NATted correctly i.e. WAN is using its external IP address.

Successful connection (using default allow rule):

Time	Flags	Connection
+0.000000	[S]	(192.168.100.1) modem < WAN
+0.000512	[S.]	(192.168.100.1) modem > WAN
+0.000770	[.]	(192.168.100.1) modem < WAN
+0.001548	[P.]	(192.169.100.1) modem < WAN	⬅︎ This is the HTTP GET request

(From there on the request proceeds as normal, web UI loads.)

Failed connection (using explicit "allow IPv4 to 192.168.100.1" rule):

Time	Flags	Connection
+0.000000	[S]	(192.168.100.1) modem < WAN	⬅︎ SYN #1
+0.000558	[S.]	(192.168.100.1) modem > WAN	⬅︎ SYN-ACK to #1
+0.000654	[S.]	(192.168.0.101) modem < WAN (192.168.100.1)	⬅︎ echo back to WAN
+0.250398	[S]	(192.168.100.1) modem < WAN	⬅︎ SYN #2
+0.250897	[S.]	(192.168.100.1) modem > WAN	⬅︎ SYN-ACK to #2
+0.251013	[S.]	(192.168.0.101) modem < WAN (192.168.100.1)	⬅︎ echo back to WAN
+0.993215	[S.]	(192.168.100.1) modem > WAN	⬅︎ SYN-ACK to #1 again?
+0.993305	[S.]	(192.168.0.101) modem < WAN (192.168.100.1)	⬅︎ echo back to WAN
+1.002322	[S]	(192.168.100.1) modem < WAN	⬅︎ SYN #3
+1.002670	[.]	(192.168.100.1) modem > WAN	⬅︎ ACK from modem?
+1.002703	[.]	(192.168.0.101) modem < WAN (192.168.100.1)	⬅︎ echo back to WAN
+1.243207	[S.]	(192.168.100.1) modem > WAN	⬅︎ SYN-ACK to #2 again?
+1.243259	[S.]	(192.168.0.101) modem < WAN (192.168.100.1)	⬅︎ echo back to WAN
+1.251912	[S]	(192.168.100.1) modem < WAN	⬅︎ SYN #4

...etc.

So – seems that packets for 192.168.0.101 that should go to LAN are going out of WAN instead.

The question remains, why?

[Fright]

(192.168.0.101) modem < WAN (192.168.100.1)   ⬅︎ echo back to WAN

can you post your rule? but not from gui, from firewall-diagnostics-pfinfo
looks like "reply-to" directive in rule
or try to disable "reply-to" in rule Advanced Options

[eth0]

Yes, checking "disable reply-to" in advanced options works.

Why is this, and is it expected behavior that I just need to know?

Original, failing rule (xx.xx.64.1 is the WAN gateway):

Code Select Expand

pass out log quick on igb2 reply-to (igb2 xx.xx.64.1) inet from any to <EPC3825:1> flags S/SA keep state label "443abb385c2b672f5c64730fc153cf92"


Working rule:

Code Select Expand

pass out log quick on igb2 inet from any to <EPC3825:1> flags S/SA keep state label "f5a961e4aa1ddf8154b2ac3ee1fdefa0"


or (in its final form):

Code Select Expand

pass out log quick on igb2 inet proto tcp from any to <EPC3825:1> port = http flags S/SA keep state label "5b8966dad703ecedd33086ba2377aaf7"


[Fright]

Yes, checking "disable reply-to" in advanced options works.

good!

Why is this, and is it expected behavior that I just need to know?

its expected behavior for pf (read man for reply-to directive).
opnsense wizard add this directive to force symmetric routing i think.
probably it makes sense to ask the devs if wizard shouldn't add this directive for "out" rules by default