Messages

Some more weirdness with this. I have since added more VLANs, and now I've seen a couple of times multicast packets from VLAN10 hosts logged as if coming from VLAN20.

The first time I noticed this I didn't have time to look into it, and the next time the lines had already been pushed out of the plain view log buffer by the time I looked, so I don't have more details about this.

In addition I still get multicasts from VLAN10 logged as denied on the disabled LAN bridge, but this VLAN20 thing was new.

Not sure how I could investigate this further. I think I'll just remove the currently unused LAN bridge, and that will take care of the spurious logging at least, but I'd still like to understand what's happening. It bothers me if a firewall behaves in ways I don't understand or expect.

yes. PA packets are "out of state" packets.
droped client packets looks like tcp retransmissions on closed connection.
then the client realized that the server would not respond and sent RA

Hm. Ok, this makes sense, thanks for the explanation. I'll try to double check this in server side logs if I still see it again.

technically its not the same connection. again, (by deafult) if you see any packet (allowed or droped) in pf log - its new connection attempt

I'm not sure if I understood now – it was a new connection that I opened after the previous one hanged, but (unless I did something strange without realizing it) it was still a connection from vlan10 to vlan20. But the firewall logged it as if vlan20 was the initiator.

...Also, while looking at the logs of a new connection after that, I see one packet also blocked in the opposite direction:

Code Select Expand

2020-11-28T19:49:31 filterlog[58643] 20,,,0,igb1_vlan20,match,block,in,4,0x10,,64,15604,0,DF,6,tcp,280,10.10.20.100,192.168.0.104,22,51534,228,PA,2589767952:2589768180,2806836895,501,,nop;nop;TS

That was just one packet though (and the connection survived it).

I think I may have been seeing something similar. Sometimes a connection to [somewhere]:443 would get randomly blocked, sometimes also :993. I don't think I had edited firewall rules, at least not every time, and the IMAPS connections likely not closed for timeout either (if I understand correctly what kind of timeout was meant here).

However today I also saw an SSH connection between my VLANs hang, with logs looking like this:

Code Select Expand


2020-11-28T19:10:55 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,92,192.168.0.104,10.10.20.100,51266,22,52,PA,741519746:741519798,2878760399,65535,,
2020-11-28T19:10:55 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,328,192.168.0.104,10.10.20.100,51266,22,288,PA,741519458:741519746,2878760399,65535,,
2020-11-28T19:10:54 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,76,192.168.0.104,10.10.20.100,51266,22,36,PA,741519710:741519746,2878760399,65535,,
2020-11-28T19:10:54 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,76,192.168.0.104,10.10.20.100,51266,22,36,PA,741519674:741519710,2878760399,65535,,
2020-11-28T19:10:54 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,76,192.168.0.104,10.10.20.100,51266,22,36,PA,741519638:741519674,2878760399,65535,,
2020-11-28T19:10:54 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,220,192.168.0.104,10.10.20.100,51266,22,180,PA,741519458:741519638,2878760399,65535,,
2020-11-28T19:10:54 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,76,192.168.0.104,10.10.20.100,51266,22,36,PA,741519602:741519638,2878760399,65535,,
2020-11-28T19:10:53 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,184,192.168.0.104,10.10.20.100,51266,22,144,PA,741519458:741519602,2878760399,65535,,
2020-11-28T19:10:52 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,184,192.168.0.104,10.10.20.100,51266,22,144,PA,741519458:741519602,2878760399,65535,,
2020-11-28T19:10:52 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,184,192.168.0.104,10.10.20.100,51266,22,144,PA,741519458:741519602,2878760399,65535,,
2020-11-28T19:10:52 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,76,192.168.0.104,10.10.20.100,51266,22,36,PA,741519566:741519602,2878760399,65535,,
2020-11-28T19:10:52 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,76,192.168.0.104,10.10.20.100,51266,22,36,PA,741519530:741519566,2878760399,65535,,
2020-11-28T19:10:52 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,112,192.168.0.104,10.10.20.100,51266,22,72,PA,741519458:741519530,2878760399,65535,,
2020-11-28T19:10:52 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,76,192.168.0.104,10.10.20.100,51266,22,36,PA,741519494:741519530,2878760399,65535,,
2020-11-28T19:10:52 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,76,192.168.0.104,10.10.20.100,51266,22,36,PA,741519458:741519494,2878760399,65535,,
2020-11-28T19:10:52 filterlog[58643] 20,,,0,igb1_vlan10,match,block,in,4,0x48,,64,0,0,DF,6,tcp,76,192.168.0.104,10.10.20.100,51266,22,36,PA,741519458:741519494,2878760399,65535,,


This went on for a while, until after a minute or so the connection was closed with rst-ack (RA).

Not sure if this is the same issue, but I don't see how state should have been reset/lost in this case. I wonder what's up here? I note the TCP sequence number/ack fields seem to have several numbers and combinations repeating in some pattern, but I don't know how to interpret that. Does this give any clue?

This could of course be caused by something in the machine I'm connecting to, but haven't seen it before.

Actually, the same thing just happened again while I was typing this. Running 20.7.4

[Original, failing rule (xx.xx.64.1 is the WAN gateway):]

Yes, checking "disable reply-to" in advanced options works.

Why is this, and is it expected behavior that I just need to know?

Original, failing rule (xx.xx.64.1 is the WAN gateway):

Code Select Expand

pass out log quick on igb2 reply-to (igb2 xx.xx.64.1) inet from any to <EPC3825:1> flags S/SA keep state label "443abb385c2b672f5c64730fc153cf92"


Working rule:

Code Select Expand

pass out log quick on igb2 inet from any to <EPC3825:1> flags S/SA keep state label "f5a961e4aa1ddf8154b2ac3ee1fdefa0"


or (in its final form):

Code Select Expand

pass out log quick on igb2 inet proto tcp from any to <EPC3825:1> port = http flags S/SA keep state label "5b8966dad703ecedd33086ba2377aaf7"


[In the tables below "modem" and "WLAN" represent MAC addresses, and when not noted, the connection is NATted correctly i.e. WAN is using its external IP address.]

I did some more digging in the capture log. I was wrong before, it's not the cable modem trying to send packets to a LAN address directly. What seems to be happening is that OPNsense echoes packets from the cable modem back to WAN, with a LAN destination address.

To understand this I tried to make a table out of the relevant data. I'm not sure that I got everything right, but I think the following is roughly what happens. In the tables below "modem" and "WLAN" represent MAC addresses, and when not noted, the connection is NATted correctly i.e. WAN is using its external IP address.

Successful connection (using default allow rule):

Time	Flags	Connection
+0.000000	[S]	(192.168.100.1) modem < WAN
+0.000512	[S.]	(192.168.100.1) modem > WAN
+0.000770	[.]	(192.168.100.1) modem < WAN
+0.001548	[P.]	(192.169.100.1) modem < WAN	⬅︎ This is the HTTP GET request

(From there on the request proceeds as normal, web UI loads.)

Failed connection (using explicit "allow IPv4 to 192.168.100.1" rule):

Time	Flags	Connection
+0.000000	[S]	(192.168.100.1) modem < WAN	⬅︎ SYN #1
+0.000558	[S.]	(192.168.100.1) modem > WAN	⬅︎ SYN-ACK to #1
+0.000654	[S.]	(192.168.0.101) modem < WAN (192.168.100.1)	⬅︎ echo back to WAN
+0.250398	[S]	(192.168.100.1) modem < WAN	⬅︎ SYN #2
+0.250897	[S.]	(192.168.100.1) modem > WAN	⬅︎ SYN-ACK to #2
+0.251013	[S.]	(192.168.0.101) modem < WAN (192.168.100.1)	⬅︎ echo back to WAN
+0.993215	[S.]	(192.168.100.1) modem > WAN	⬅︎ SYN-ACK to #1 again?
+0.993305	[S.]	(192.168.0.101) modem < WAN (192.168.100.1)	⬅︎ echo back to WAN
+1.002322	[S]	(192.168.100.1) modem < WAN	⬅︎ SYN #3
+1.002670	[.]	(192.168.100.1) modem > WAN	⬅︎ ACK from modem?
+1.002703	[.]	(192.168.0.101) modem < WAN (192.168.100.1)	⬅︎ echo back to WAN
+1.243207	[S.]	(192.168.100.1) modem > WAN	⬅︎ SYN-ACK to #2 again?
+1.243259	[S.]	(192.168.0.101) modem < WAN (192.168.100.1)	⬅︎ echo back to WAN
+1.251912	[S]	(192.168.100.1) modem < WAN	⬅︎ SYN #4

...etc.

So – seems that packets for 192.168.0.101 that should go to LAN are going out of WAN instead.

The question remains, why?

Doesn't that depend on whether it's defined as an "in" rule or an "out" rule?

Both the explicit rule that I tried to create, and the default floating rule that gets applied if I remove my own rule, are evaluated on WAN (as "out" rules). I can see this in the log, and I can see in packet capture that actual packets are sent and received.

Certainly you also need to have an "in" rule on the LAN to allow packets into the firewall on the browser end of things, but this is taken care of in the default rules (allow LAN to any), otherwise there wouldn't be anything to match or capture on the WAN side I think.

Or am I missing something really fundamental here?

Another newbie question...

I'm trying to access a bridged cable modem's web UI that responds at 192.168.100.1:80 on the WAN interface. This works fine using the default Floating rule "let out anything from firewall host itself" (pass IPv4+6 to any).

However if I define my own WAN firewall rule (pass IPv4 to 192.168.100.1 only), so I could also block any other private addresses from leaking out on WAN, the browser can no longer load the UI (server not responding). There's nothing else blocking it, I have turned on logging on all WAN and Floating rules.

I can't view the actual definition of the default rule beyond what's shown in the firewall rules list, but on that level I don't see any difference to my own rule.

Could this be some kind of a gateway issue, as the modem's IP is outside of the WAN interface's subnet?

Doing a packet capture on WAN with the default rule, I see that communication happens between the firewall host's WAN address and 192.168.100.1, and the session soon proceeds into HTTP headers and such. With my own rule, after each response from 192.168.100.1 to the firewall host, it sends (or tries) another response to 192.168.0.101 (the laptop where I'm running the browser), and this repeats until the browser times out. I don't know why this is happening.

I think at some point I was able to make the default rule fail by changing some gateway related setting, but can't recall what it was and can't find it again. (I could be wrong and it may have failed for some unrelated reason.)

I've also tried changing the Gateway setting in my own rule but it won't let me (Policy based routing is only supported on inbound rules).

Is there some special magic going on in the default rule that I'm missing?

[not enabled]

Ok, I'm new with this, might be a simple question (I hope!) but I can't figure this out.

I have the following interfaces:

LAN1 (igb1)
VLAN10 (igb1_vlan10), with parent LAN1
LAN (bridge0), with member interface LAN1, not enabled

The bridge is going to include LAN0 (igb0) later. VLAN10 was originally created on the bridge, but I moved (re-parented) it to igb1 directly while solving some other problem, and also unchecked "Enable Interface" on the bridge to take it out of the equation completely.

This is mostly working fine (I have seen some weirdness which I may come back to later), but the disabled LAN bridge is showing traffic in the firewall live view:

⊘

VLAN10

➔

Nov 21 20:38:06

[fe80::18ad:c22e:c697:a75e]:5353

[ff02::fb]:5353

udp

Default deny rule

⊘

LAN

➔

Nov 21 20:38:06

[fe80::18ad:c22e:c697:a75e]:5353

[ff02::fb]:5353

udp

Default deny rule

⊘

LAN

➔

Nov 21 20:38:06

192.168.0.105:5353

224.0.0.251:5353

udp

Default deny rule

⊘

VLAN10

➔

Nov 21 20:38:04

[fe80::25:7c21:b051:7bd7]:5353

[ff02::fb]:5353

udp

Default deny rule

⊘

LAN

➔

Nov 21 20:38:04

[fe80::25:7c21:b051:7bd7]:5353

[ff02::fb]:5353

udp

Default deny rule

⊘

VLAN10

➔

Nov 21 20:38:01

[fe80::25:7c21:b051:7bd7]:5353

[ff02::fb]:5353

udp

Default deny rule

⊘

LAN

➔

Nov 21 20:38:01

[fe80::25:7c21:b051:7bd7]:5353

[ff02::fb]:5353

udp

Default deny rule

⊘

VLAN10

➔

Nov 21 20:38:00

[fe80::25:7c21:b051:7bd7]:5353

[ff02::fb]:5353

udp

Default deny rule

⊘

LAN

➔

Nov 21 20:38:00

[fe80::25:7c21:b051:7bd7]:5353

[ff02::fb]:5353

udp

Default deny rule

⊘

LAN

➔

Nov 21 20:38:00

192.168.0.106:5353

224.0.0.251:5353

udp

Default deny rule

⊘

LAN

➔

Nov 21 20:37:28

192.168.0.101:64657

239.255.255.250:1900

udp

Default deny rule

⊘

VLAN10

➔

Nov 21 20:37:18

[fe80::18ad:c22e:c697:a75e]:5353

[ff02::fb]:5353

udp

Default deny rule

First I thought maybe the connected switch is misbehaving and sending non-tagged frames, which might somehow end up in the wrong place, but soon realized that the same IPv6 multicast packets are also logged on VLAN10 (while IPv4 multicast only gets logged on LAN) – so they must be tagged. I still confirmed this by doing a packet capture on igb1, and couldn't see anything other than vlan10 traffic.

The difference between IPv4 and IPv6 is probably because at the moment I have "IPv6 Configuration Type: None" on VLAN10, but that's not the question I have here.

The actual question I have is: Why does tagged VLAN10 traffic go to the bridge which is not part of that VLAN, *AND* why does anything at all get sent to and logged on the LAN bridge when it's not enabled?

I could of course nuke the bridge and re-create it when I need it, and that would make the problem go away, but I'd like to understand why it's behaving the way it is.

[Edit: Figured out how to (ab)use BBCode to insert logs inline. Screenshot image still attached below, but seems it's not visible if not logged in to the forum.]