Trouble with half-closed connections

[Crazyachmed]

I investigated an unusually high number of packets hitting my default deny rule. For the most part they had the TCP-FPA flags set. Looking at a trace I did on the box hosting the firewall-VM I looked at some affected connections and the most unusual about them is that the client still sends traffic into a half-closed connection. Two examples:

1) Server sends FIN
2) Client ACKs the FIN
3) Client Sends additional data -> Packet is blocked
4) Client sends FIN -> Packet is blocked
5) Client retransmits FIN a few times -> Packets are blocked

Second case:
1) Server sends FIN
2) Client ACKs packet before FIN
3) Server retransmits FIN
4) Client ACKs the FIN
5) Client sends FIN -> Packet is blocked
6) Client retransmits FIN a few times -> Packets are blocked

Both clients are Android devices talking to the cloud. One connection is IPv4, the other is IPv6. How can I troubleshoot this further? Why don't the packets match the stateful return path?

[esquagga]

I haven't sniffed the traffic yet, but I'm wondering if this is the same thing I am seeing. I see a lot of blocks to normal ports. I put multiple rules in to pass the traffic just to be sure. Still happens.

I'm running 20.7.3-amd64.

[Crazyachmed]

I haven't sniffed the traffic yet, but I'm wondering if this is the same thing I am seeing.

Can you please add the following three floating rules at the bottom of the list of floating rules in this order?

• Action: Block
• Quick: Disabled
• TCP/IP Version: IPv4+IPv6
• Protocol: TCP
• Log: Enable
• Description: Stale TCP
• Advanced, TCP flags, ACK: Both checkboxes

• Action: Block
• Quick: Disabled
• TCP/IP Version: IPv4+IPv6
• Protocol: TCP
• Log: Enable
• Description: Stale TCP FIN
• Advanced, TCP flags, FIN: Both checkboxes

• Action: Block
• Quick: Disabled
• TCP/IP Version: IPv4+IPv6
• Protocol: TCP
• Log: Enable
• Description: Stale TCP RST
• Advanced, TCP flags, RST: Both checkboxes

This way you can easily see what kind of packets hit the "Default deny rule" without looking at each entry's details. In my case I see a lot of hits for FIN.

On another note I completely disabled all HW-offloading on the VM-Host using this blunt instrument:

Code: [Select]

#!/bin/bash

OFFOPTS="rx tx tso ufo gso gro lro rxvlan txvlan rxhash ntuple"
INTF=$(/sbin/ifconfig | /bin/egrep '^\S' | /usr/bin/cut -d':' -f 1 | /usr/bin/tr '\n' ' ')

function disoff {
  for OPTION in $OFFOPTS; do
    /sbin/ethtool --offload "$1" "$OPTION" off &>/dev/null || true
  done
}

for CUR in $INTF; do
  disoff $CUR
done

Sadly, there was no improvement

[mimugmail]

Havent followed the thread but async routing usually comes in interfaces with upstream gateway set and also a second device in this network

[Crazyachmed]

Havent followed the thread but async routing usually comes in interfaces with upstream gateway set and also a second device in this network

I only have the one gateway configured. A few client networks and one WAN-network.

Also looking at the trace I do see all packets up until the client packet dropped on the firewall itself.

[Crazyachmed]

I have filed a bug report: https://github.com/opnsense/core/issues/4391

[mimugmail]

Is there an option to test this with 20.1?

[Fright]

hm..why does it take 19-29 minutes (!) to send FA after A from client?
of course the connection has been closed for a long time

[Crazyachmed]

Looking into this further I saw a case where it took 8:40h (!) for the client to close the connection after the server closed it.

I experimented with setting tcp.closing to 3600s (originally 900s), which of course makes some of the connections work. But setting a very long timeout does not help, becacues by then the server has already closed its socket and only sends RSTs back.

Currently there is also no method of permanently modifying any session timeouts outside of the provided templates. For completness sake, this is the very temporary workaround:

• Add "set timeout tcp.closing 3600" to /tmp/rules.debug
• Apply using "pfctl -f /tmp/rules.debug"

Which can only be implemented via a feature request, if anyone has a need for someting like this.[/list]