[solved] Rule weirdness / Allow all not matching

[mooh]

I have a network that's supposed to have unrestricted IPv4 internet access. The rule set has worked fine for years. Today, I was made aware that some Apple iCloud service seems to be broken. Sure enough, with global logging on, I saw that connections to 17.145.16.2:443 were blocked by the default rule. While scratching my head trying to find out what went wrong, I saw that the pass rule kicked in again without any intervention.

The screenshot shows 2 source devices, but the pass rule doesn't care about the source address. No one else had access to the firewall.

What could possibly be the reason for this behaviour?

System is OPNsense 26.4_14-amd64

[mooh]

The rules

[lmoore]

Best to start by looking at the detailed information in the log (https://docs.opnsense.org/manual/logging_firewall.html#live-view), which is at the end of the line.

Click on the information icon. In the window which opens, scroll down to the end and and look for the tcpflags line. You may see flags of F or R in the logs for the blocked connections, as they all relate to the connection which logged a pass at 14:45:23 - which should show a flag of S.

It may have been the connection was closed but the client belatedly sent flags to close the connection - judging by the time stamps.

Did you check the Apple System Status page at that time?

[mooh]

Thanks a lot for your suggestion. I just got myself a refresher on TCP flags...

The flags in the logs seem support the theory that the pass rule didn't apply because the connection state was lost. The green lines in the log have state SEC, followed by the sequence of default block messages with flags FPA, except the last one with flags RA. If my understanding is correct, this indicates that the client wanted to shut down the connection for a while, not getting anywhere and finally giving up with a reset.

Is there a way to see why a firewall state may expire while a client still thinks it is active? I'm pretty sure it's not because the firewall state were exhausted. Currently, it only uses ~500 of ~800000 and I don't see why the load could have been that much larger earlier on.

[lmoore]

Is there a way to see why a firewall state may expire while a client still thinks it is active?

The client would have sent a packet with the TCP flag FIN set and pf will have seen this and set the timers to close the state. For whatever reason, the client didn't consider the connection fully closed and sent subsequent packets with FIN set and then at the end a packet with RST set. The log entries were for out-of-state connection attempts and is why they were blocked.

I expect someone on the list can provide a better explanation regarding this.

To see what is happening you could SSH to your firewall and use 'tcpdump', with the '-p' option set, to capture and save a session. Once the session has been captured, including the out-of-state packets, you can review it in a tool such as Wireshark and observe when the flags get set and when packets are re-transmitted.

Of course, what you were looking at is a red herring as none of this explains why your clients were experiencing connectivity issues to iCloud.

[Edit] To obtain the pf state timeouts for TCP sessions on your firewall, run the following;

  pfctl -s timeouts | grep tcp

    tcp.first                      120s
    tcp.opening                     30s
    tcp.established              86400s
    tcp.closing                    900s
    tcp.finwait                     45s
    tcp.closed                      90s
    tcp.tsdiff                      30s

In addition, you could also run 'pftop' and watch the sessions live;

/usr/local/sbin/pftop -s1 -f 'net 192.168.150 and host 17.145.16.2 and proto tcp and port 443'

[mooh]

Thanks for your support. I believe the "iCloud not working" report and the observation made in the firewall log at that time are just a coincident. The device trying to close the connection is HomePod Mini. I need to find out what exactly didn't work work.