TCP MSS - Firewall: Settings: Normalization

[danderson]

Anyone else seeing/noticing issues with MSS? I have had my MSS set to 1300 for IPSEC and WG for years and it has been working well, but after the 24.1 update (including 24.1.1) its either not working or something else is going on, UDP I get full speed, but TCP very slow like alot of frag. Ive even tried lowering MSS to 1260 to no effect.  I can see in my transport graphs that this changed on 1/30/24 with the update to 24.1.



# iperf3 -c X.X.X.X -b 950M
Connecting to host X.X.X.X, port 5201
[  5] local X.X.X.X port 34276 connected to X.X.X.X port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.25 MBytes  10.5 Mbits/sec   16   9.75 KBytes
[  5]   1.00-2.00   sec   867 KBytes  7.11 Mbits/sec   10   13.4 KBytes
[  5]   2.00-3.00   sec   669 KBytes  5.48 Mbits/sec   12   12.2 KBytes
[  5]   3.00-4.00   sec   726 KBytes  5.94 Mbits/sec   16   6.09 KBytes
[  5]   4.00-5.00   sec   634 KBytes  5.19 Mbits/sec   13   9.75 KBytes
[  5]   5.00-6.00   sec   760 KBytes  6.23 Mbits/sec   17   6.09 KBytes
[  5]   6.00-7.00   sec   824 KBytes  6.75 Mbits/sec   13   12.2 KBytes
[  5]   7.00-8.00   sec   768 KBytes  6.29 Mbits/sec   17   4.88 KBytes
[  5]   8.00-9.00   sec   640 KBytes  5.24 Mbits/sec   15   6.09 KBytes
[  5]   9.00-10.00  sec   620 KBytes  5.08 Mbits/sec   15   3.66 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.61 MBytes  6.38 Mbits/sec  144             sender
[  5]   0.00-10.01  sec  7.43 MBytes  6.23 Mbits/sec                  receiver

iperf Done.
iperf3 -c X.X.X.X -b 950M -u
Connecting to host X.X.X.X, port 5201
[  5] local X.X.X.X port 58862 connected to X.X.X.X port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  77.4 MBytes   649 Mbits/sec  65016
[  5]   1.00-2.00   sec   114 MBytes   957 Mbits/sec  95900
[  5]   2.00-3.00   sec   118 MBytes   987 Mbits/sec  98870
[  5]   3.00-4.00   sec   113 MBytes   947 Mbits/sec  94853
[  5]   4.00-5.00   sec   114 MBytes   959 Mbits/sec  96052
[  5]   5.00-6.00   sec   115 MBytes   968 Mbits/sec  97000
[  5]   6.00-7.00   sec   115 MBytes   961 Mbits/sec  96237
[  5]   7.00-8.00   sec   104 MBytes   876 Mbits/sec  87765
[  5]   8.00-9.00   sec   116 MBytes   975 Mbits/sec  97616
[  5]   9.00-10.00  sec   117 MBytes   984 Mbits/sec  98529
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.08 GBytes   926 Mbits/sec  0.000 ms  0/927838 (0%)  sender
[  5]   0.00-10.02  sec   646 MBytes   541 Mbits/sec  0.012 ms  381658/924217 (41%)  receiver

[mimugmail]

There is a change to enable scrub for All ifs but it shouldnt be in stable yet. Can you try disble it for ipsec in normalization?

[danderson]

There is a change to enable scrub for All ifs but it shouldnt be in stable yet. Can you try disble it for ipsec in normalization?

I removed the IPSEC if and the VTI ifs with no changes, left it on LAN ifs.  Also tried disabling the rule as a whole under normalization but that obv would make frags over VTI/IPSEC as i dont have MSS set on the physical ifs under interfaces.

[newsense]

Hey danderson, weird dinosaur you have awaken :)

If I recall correctly the low MTU on the WAN sowed quite a few releases back.


How are your GUI WAN settings looking from MTU section to the bottom ? And can you post the console WAN information please -- excluding IPs/MAC which are not relevant.

Code Select Expand

ifconfig em0/igb0/vtnet0

[danderson]

Newsense,

Nothing special going on here, MTU all blank aka default, i see on the console that its 1500 and each IPSEC is 1400. Always had v4 &v6 VTI and set MSS in normalization to 1300.  Strange as well as it seems to be only 1 direction on the tunnel, the reverse direction of the tunnel seems to be normal.  Ive tested the provider and I get full 1g on speedtests, again same as you can see above with the UDP iperf.  Im at a loss on it.

ifconfig igb0
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN (wan)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
        ether xx:xx:xx:xx:xx:xx
        inet X.X.X.X netmask 0xfffffffc broadcast x.x.x.x
        inet6 xxxx:xxxx:xxxx::xxxx prefixlen 126
        inet6 fe80::xxxx:xxxx::xxxx%igb0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>


lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
enc0: flags=41<UP,RUNNING> metric 0 mtu 1536
ipsec1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400

[newsense]

Yeah you seem to be OK MTU wise, was wondering if there's a  discrepancy between GUI and CLI.


When I looked today on a FW the HW override was checked but no fields were populated, and the WAN MTU looked like this before doing the override properly again:

Code Select Expand

igc0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 576
        description: WAN (wan)
        options=4e427bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>





Even with the catastrophic mtu set at 576 by the ISP, on Anyconnect (so DTLS) and Wireguard VPNs things were a lot more manageable than trying to refresh a page in a browser - and the pattern remained even after the mtu was fixed.


I'll try later to install iperf3 on a couple FW4Cs and see if I can get similar results on UDP vs TCP

[danderson]

https://github.com/opnsense/core/issues/7203

https://github.com/opnsense/core/commit/630ab193b6965a3dabb0c43a3495dba16cd399ab

I am going to test the change that mimugmail stated above, easy to revert if needed.

opnsense-patch 630ab19

[newsense]

Thanks for the reminder, I had it applied briefly before 24.1.1 and seemed fine

[danderson]

Thanks for the reminder, I had it applied briefly before 24.1.1 and seemed fine

ya no issues with the patch here, but still no change for me on TCP performance over the tunnel.

[danderson]

newsense

fixed my issue, strangest thing.   Anyways, lots of troubleshooting over here, LAGG removing all but 1 port at a time, flow control on the switch setting to off, restarting supervisor in core 1 and moved sup to core 2.  Saw today in health reporting on both sides that the gateway that the VTI routes over that it was dropping packets, unknown why as Internet circuit and monitoring wasnt dropping any packets.

I had been using aes256gcm16-sha512-ecp521 (DH21 NIST EC) for awhile on P1 & P2. decided to mess with the all of that and just use default in opnsense in P1 & P2. 

As soon as the tunnel reconnected, my TCP performance went back to normal and im getting my full 1G over the VTI.  I'm happy its fixed as my SAN replication can now catchup on the few T of replication it was behind.

Also no more drops over IPSEC.   Take it as you will, but im burnt on trying things over the last week, will let it run for awhile and maybe mess more later with specific encryption and ciphers.

[danderson]

maybe spoke too soon, I got full speed for like an hour, now its back to being slow again.   Still troubleshooting arghhhhh

[newsense]

Given the latest info...unsure if this is fixable in software, seems coincidental.

[danderson]

Given the latest info...unsure if this is fixable in software, seems coincidental.

It is coincidental.  There is / was an issue with the upstream provider that the colo is working on. What a #$%$