Slow downloads on some connections

[timc]

I have just installed OPNsense 24.1.5_3-amd64 on a machine migrating from OpenWRT x86.

My connection is a typical Dutch PPPoE internet connection over VLAN6 with 1 Gigabit speeds. The internet provider supports an MTU of 1500 within the PPPoE tunnel. The hardware contains a Intel i226-V NIC (connected to WAN) and a Intel X710 NIC (connected to LAN).

The issue I'm seeing is that downloads are slow. For example I'm only getting +/- 7 MB/s on http://speedtest.tele2.net/ and only 3 MB/s on https://speedtest.serverius.net/. However I'm able to reach almost full gigabit speeds on speedtest.net on the same hardware. The downloads from the speedtest servers are full speed again when switching back to the OpenWRT machine.

I did a packet-dump and noticed that a TCP MSS of 1452 was being returned on the TCP connection when downloading from the speedtest servers.

However when doing ping tests I'm able to successfully send pings with an MTU of 1500 and receive a reply.

I'm out of options how to fix this. Do you have any ideas?

[meyergru]

I have a similar configuration and no problems with speed from both sites. I see an MSS of 1440, although I am sure I have configured my MTU to allow for 1500 bytes.

I found that the recent optimizations for MTU calculation did not work for me, so I had to tweak it manually, maybe these settings will help you:

On the physical parent ethernet interface, I have set an MTU of 1512 (I define this anyway to access my ONT UI), on the VLAN, you cannot set it and on the pppoe interface, I have set MTU = 1508. I leave the MSS at defaults, as well as the LAN settings.

This results in these MTUs:

Code Select Expand


root@OPNsense:~ # ifconfig igc3
igc3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1512
        description: ONT (opt15)
        options=4e0272b<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,TSO4,TSO6,LRO,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 00:e2:69:5c:55:55
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
        groups: LOCAL_VLANS
        media: Ethernet autoselect (2500Base-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@OPNsense:~ # ifconfig igc3_vlan40
igc3_vlan40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1508
        options=4000000<NOMAP>
        ether 00:e2:69:5c:55:55
        groups: vlan
        vlan: 40 vlanproto: 802.1q vlanpcp: 0 parent interface: igc3
        media: Ethernet autoselect (2500Base-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@OPNsense:~ # ifconfig pppoe0
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN (wan)
        inet6 fe80::2e2:69ff:fe5c:5555%pppoe0 prefixlen 64 scopeid 0x13
        inet6 fe80::2e2:69ff:fe5c:5554%pppoe0 prefixlen 64 scopeid 0x13
        inet 99.100.101.102 --> 81.133.17.26 netmask 0xffffffff
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>


With these settings, I can really use all of the 1500 bytes MTU. Keep in mind that you probably need a reboot to apply them correctly, because the order of assigments seem to matter (as you can see, the resulting pppoe MTU is not 1508 as instructed, don't ask me why).



Other than that, do you see any packet errors on your interfaces?

[schmuessla]

1452 looks quite correct (1500 Ethernet - 8 PPPoE - 20 TCP - 20 IPv4).
However speed would not be too bad if it would be a couple of bytes too low, only too high is an issue.

With OpenWRT you get other speeds or are they the same?

[timc]

Thank you for your replies!

I'm sure the MTU of the PPPoE line is configured correctly (1500 internal, 1508 on the VLAN, 1512 on the bare interface) because I'm able to ping with 1500-byte packets successfully (when I revert to the "standard" 1492 I can't as expected).

The issue seems to be with some TCP connections that show a MSS of 1452 where 1460 would be expected (1508 Ethernet).

So at the moment:
When using OpenWRT: Speedtest.net shows 1 gigabit up/down, downloads of speedtest-files from telenet.net, serverius.net (and others) are (almost) gigabit speeds, like 900 megabit/s+
When using OPNsense: Speedtest.net shows 1 gigabit up/down, downloads of speedtest-files are only 20 megabit/s.

And 1500-size pings work on both setups.

[bestboy]

The issue seems to be with some TCP connections that show a MSS of 1452 where 1460 would be expected [...]

When using OPNsense: [...] downloads of speedtest-files are only 20 megabit/s.

Not an expert, but I don't see how a difference of 8 bits in maximum segment size will cost you a whopping 800 MBit/s in throughput. I don't think looking at MTU or MSS is going to reveal the issue at hand.