Anyone else seeing/noticing issues with MSS? I have had my MSS set to 1300 for IPSEC and WG for years and it has been working well, but after the 24.1 update (including 24.1.1) its either not working or something else is going on, UDP I get full speed, but TCP very slow like alot of frag. Ive even tried lowering MSS to 1260 to no effect. I can see in my transport graphs that this changed on 1/30/24 with the update to 24.1.
# iperf3 -c X.X.X.X -b 950M
Connecting to host X.X.X.X, port 5201
[ 5] local X.X.X.X port 34276 connected to X.X.X.X port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec 16 9.75 KBytes
[ 5] 1.00-2.00 sec 867 KBytes 7.11 Mbits/sec 10 13.4 KBytes
[ 5] 2.00-3.00 sec 669 KBytes 5.48 Mbits/sec 12 12.2 KBytes
[ 5] 3.00-4.00 sec 726 KBytes 5.94 Mbits/sec 16 6.09 KBytes
[ 5] 4.00-5.00 sec 634 KBytes 5.19 Mbits/sec 13 9.75 KBytes
[ 5] 5.00-6.00 sec 760 KBytes 6.23 Mbits/sec 17 6.09 KBytes
[ 5] 6.00-7.00 sec 824 KBytes 6.75 Mbits/sec 13 12.2 KBytes
[ 5] 7.00-8.00 sec 768 KBytes 6.29 Mbits/sec 17 4.88 KBytes
[ 5] 8.00-9.00 sec 640 KBytes 5.24 Mbits/sec 15 6.09 KBytes
[ 5] 9.00-10.00 sec 620 KBytes 5.08 Mbits/sec 15 3.66 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 7.61 MBytes 6.38 Mbits/sec 144 sender
[ 5] 0.00-10.01 sec 7.43 MBytes 6.23 Mbits/sec receiver
iperf Done.
iperf3 -c X.X.X.X -b 950M -u
Connecting to host X.X.X.X, port 5201
[ 5] local X.X.X.X port 58862 connected to X.X.X.X port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 77.4 MBytes 649 Mbits/sec 65016
[ 5] 1.00-2.00 sec 114 MBytes 957 Mbits/sec 95900
[ 5] 2.00-3.00 sec 118 MBytes 987 Mbits/sec 98870
[ 5] 3.00-4.00 sec 113 MBytes 947 Mbits/sec 94853
[ 5] 4.00-5.00 sec 114 MBytes 959 Mbits/sec 96052
[ 5] 5.00-6.00 sec 115 MBytes 968 Mbits/sec 97000
[ 5] 6.00-7.00 sec 115 MBytes 961 Mbits/sec 96237
[ 5] 7.00-8.00 sec 104 MBytes 876 Mbits/sec 87765
[ 5] 8.00-9.00 sec 116 MBytes 975 Mbits/sec 97616
[ 5] 9.00-10.00 sec 117 MBytes 984 Mbits/sec 98529
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.08 GBytes 926 Mbits/sec 0.000 ms 0/927838 (0%) sender
[ 5] 0.00-10.02 sec 646 MBytes 541 Mbits/sec 0.012 ms 381658/924217 (41%) receiver
There is a change to enable scrub for All ifs but it shouldnt be in stable yet. Can you try disble it for ipsec in normalization?
Quote from: mimugmail on February 06, 2024, 06:09:34 PM
There is a change to enable scrub for All ifs but it shouldnt be in stable yet. Can you try disble it for ipsec in normalization?
I removed the IPSEC if and the VTI ifs with no changes, left it on LAN ifs. Also tried disabling the rule as a whole under normalization but that obv would make frags over VTI/IPSEC as i dont have MSS set on the physical ifs under interfaces.
Hey danderson, weird dinosaur you have awaken :)
If I recall correctly the low MTU on the WAN sowed quite a few releases back.
How are your GUI WAN settings looking from MTU section to the bottom ? And can you post the console WAN information please -- excluding IPs/MAC which are not relevant.
ifconfig em0/igb0/vtnet0
Newsense,
Nothing special going on here, MTU all blank aka default, i see on the console that its 1500 and each IPSEC is 1400. Always had v4 &v6 VTI and set MSS in normalization to 1300. Strange as well as it seems to be only 1 direction on the tunnel, the reverse direction of the tunnel seems to be normal. Ive tested the provider and I get full 1g on speedtests, again same as you can see above with the UDP iperf. Im at a loss on it.
ifconfig igb0
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN (wan)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether xx:xx:xx:xx:xx:xx
inet X.X.X.X netmask 0xfffffffc broadcast x.x.x.x
inet6 xxxx:xxxx:xxxx::xxxx prefixlen 126
inet6 fe80::xxxx:xxxx::xxxx%igb0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
enc0: flags=41<UP,RUNNING> metric 0 mtu 1536
ipsec1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400
Yeah you seem to be OK MTU wise, was wondering if there's a discrepancy between GUI and CLI.
When I looked today on a FW the HW override was checked but no fields were populated, and the WAN MTU looked like this before doing the override properly again:
igc0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 576
description: WAN (wan)
options=4e427bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
Even with the catastrophic mtu set at 576 by the ISP, on Anyconnect (so DTLS) and Wireguard VPNs things were a lot more manageable than trying to refresh a page in a browser - and the pattern remained even after the mtu was fixed.
I'll try later to install iperf3 on a couple FW4Cs and see if I can get similar results on UDP vs TCP
https://github.com/opnsense/core/issues/7203
https://github.com/opnsense/core/commit/630ab193b6965a3dabb0c43a3495dba16cd399ab
I am going to test the change that mimugmail stated above, easy to revert if needed.
opnsense-patch 630ab19
Thanks for the reminder, I had it applied briefly before 24.1.1 and seemed fine
Quote from: newsense on February 07, 2024, 06:59:34 PM
Thanks for the reminder, I had it applied briefly before 24.1.1 and seemed fine
ya no issues with the patch here, but still no change for me on TCP performance over the tunnel.
newsense
fixed my issue, strangest thing. Anyways, lots of troubleshooting over here, LAGG removing all but 1 port at a time, flow control on the switch setting to off, restarting supervisor in core 1 and moved sup to core 2. Saw today in health reporting on both sides that the gateway that the VTI routes over that it was dropping packets, unknown why as Internet circuit and monitoring wasnt dropping any packets.
I had been using aes256gcm16-sha512-ecp521 (DH21 NIST EC) for awhile on P1 & P2. decided to mess with the all of that and just use default in opnsense in P1 & P2.
As soon as the tunnel reconnected, my TCP performance went back to normal and im getting my full 1G over the VTI. I'm happy its fixed as my SAN replication can now catchup on the few T of replication it was behind.
Also no more drops over IPSEC. Take it as you will, but im burnt on trying things over the last week, will let it run for awhile and maybe mess more later with specific encryption and ciphers.
maybe spoke too soon, I got full speed for like an hour, now its back to being slow again. Still troubleshooting arghhhhh
Given the latest info...unsure if this is fixable in software, seems coincidental.
Quote from: newsense on February 11, 2024, 12:19:18 AM
Given the latest info...unsure if this is fixable in software, seems coincidental.
It is coincidental. There is / was an issue with the upstream provider that the colo is working on. What a #$%$