Slow upload vs previous router

[chamley]

Hello all.  I'm new to OPNsense and could use some help!

I have a 1000/1000 Mbps FTTH internet connection, but I'm struggling to get OPNsense to perform well on the uplink.

Previously I used a Synology RT2600AC as my router.  With that, with both fast.com and speedtest.net I would get:
~ 800 Mbps down
~ 800 Mbps up

I've now switched to OPNsense 23.1.7 running on an Intel NUC with an i5-1340P, 16 GB RAM and dual Intel i226 NICs.  I now get:
~ 1000 Mbps down.  CPU utilisation is <4%
< 100 Mbps up.  CPU utilisation is ~30%

Things I've checked/tried:
- Disabling spectre and meltdown mitigations.
- Disabling IPv6.
- WAN MTU.  It's 1500, which I believe is correct for my provider.
- No intrusion detection or other packages active.
- Firewall is enabled (default configuration).
- VLANs.  I am not using any and have not changed and VLAN-related settings.
- LAN DNS and DHCP.  These are both disabled as they are provided by another device on the network.
- Hardware offloading is disabled for CRC, TSO and LRO.

[CJ]

That config should have no problem giving you line speed both ways, AFAIK.  I don't have a connection to test that with, unfortunately.

What speeds were you getting with the default install?  No changes to the config at all.

Have you tried putting the NUC between two machines locally and doing an iperf across them?

[chamley]

Hello!  I'm sorry for replying so slowly.  I missed the email notification!

I have just tried a completely default install of 22.7 and get:
~1000 Mbps down
~150 Mbps up

Back to the Synology router and I get:
~1000 Mbps down
~1000 Mbps up

Unfortunately I haven't had time to try iperf across the NUC.

[chamley]

I've just tried 23.1.9 with default setup and the results are the same, except CPU utilisation during upload is <10%.

[zan]

Are you connecting to your ISP with PPPOE? You may want to set net.isr.dispatch to deferred if you do.

[Seimus]

Additional to what @zan said. Try to change the tunnables. PPPoE behavies wierd.

OVerall your CPu should be able to do 1G on Single core without problem. However you are using a NUC, NUC usually has only a single onboard NIC.

Can you tell what NUC do you have?
How did you expand your NIC count?
What kind of adapter did you use?
Is that adapter okay?
Do you see any errors on the Interface that has the adapter?

Regards,
S.

[CJ]

Hello!  I'm sorry for replying so slowly.  I missed the email notification!

I have just tried a completely default install of 22.7 and get:
~1000 Mbps down
~150 Mbps up

Back to the Synology router and I get:
~1000 Mbps down
~1000 Mbps up

Unfortunately I haven't had time to try iperf across the NUC.

No worries about the delay.  I've been there before.  How do you have the NUC connected?  Is it set to 1g on each side?  I know a lot of people have talked about the Intel 2.5g NICs having issues.

Also, are you actually seeing 1000 Mbps or are you rounding the 900ish gig limit?  A lot of ISPs overprovision so if you're connected via 2.5g I would expect to see something like 1150 Mbps.

Additional to what @zan said. Try to change the tunnables. PPPoE behavies wierd.

OVerall your CPu should be able to do 1G on Single core without problem. However you are using a NUC, NUC usually has only a single onboard NIC.

Can you tell what NUC do you have?
How did you expand your NIC count?
What kind of adapter did you use?
Is that adapter okay?
Do you see any errors on the Interface that has the adapter?

Regards,
S.

I can't speak to the PPPoE but they stated in the OP that the NUC they're using has dual i226 2.5g NICs.  Not all minis are single NIC machines.

[Seimus]

I can't speak to the PPPoE but they stated in the OP that the NUC they're using has dual i226 2.5g NICs.  Not all minis are single NIC machines.

Yes i've seen, but its better to have it confirmed. Cause the mainstream NUC from Intel - Intel NUC 13 I am aware of usually had only 1 NIC.


Regards,
S.

[chamley]

Hello everyone, thanks for your replies.  To answer your questions:

Are you connecting to your ISP with PPPOE? You may want to set net.isr.dispatch to deferred if you do.  Yes, PPPoE.  I tried a few tuneables yesterday:  net.isr.dispatch=deferred, net.isr.bindthreads=1, net.isr.maxthreads=-1  This didn't make any difference.
EDIT: My mistake! It's not PPPoE. I have FTTP using IPoE.

Can you tell what NUC do you have?  NUC13ANHI5
How did you expand your NIC count?  This series of NUCs has a header for an additional NIC and 2x USB ports, same as previous generations of NUC that had two NICs.  I am using the official Intel module.  The USB ports are not in use.
What kind of adapter did you use?  Intel i226
Is that adapter okay?  Seems to work fine.  I have tried swapping between the two NICs and they perform the same.
Do you see any errors on the Interface that has the adapter?  No.

How do you have the NUC connected?  Is it set to 1g on each side?  Correct.
Also, are you actually seeing 1000 Mbps or are you rounding the 900ish gig limit?  900 is the average over most tests.  For the first few seconds I can see up to ~1300.  I guess the ISP is then throttling.


I've repeated more tests and the CPU usage is definitely spiking during upload.
I tried disabling the E-cores, leaving only the 4 P-cores.  During upload, I hit ~150 Mbps with 50% CPU usage  :o

[clarknova]

Run

Code Select Expand

top -PSH to show detailed CPU usage during your tests.

[CJ]

How do you have the NUC connected?  Is it set to 1g on each side?  Correct.
Also, are you actually seeing 1000 Mbps or are you rounding the 900ish gig limit?  900 is the average over most tests.  For the first few seconds I can see up to ~1300.  I guess the ISP is then throttling.

If you're seeing 1300 then you don't have the NICs set to 1g.  What does the Media say for each of the interfaces?

What devices do you have connected to the NUC?  Please list everything from the fiber convertor/modem all the way to the PC you are testing from.

[chamley]

Real life got in the way for a bit but I'm now back trying to get this working.

Both interfaces on the NUC are definitely set to 1Gbps.  I believe the brief speed test results of 1300 are a glitch caused by the way the sites are calculating the average.  It seems to be particularly common with fast.com.  The final result on these test is always ~900.

The network arrangement is:
Fibre -> ISP modem -> NUC -> Netgear switch -> PC

I normally run the Synology RT2600AC in place of the NUC, which is giving me close to line speed.  When the NUC is in place instead of the RT2600AC, the upload becomes slow.  Restarting the modem doesn't appear to make any difference.

During upload, top shows 20-30% on all 4 cores for kernel(if_io_tqg) and speed is ~150Mbps.
During download, I get ~10% on all cores for kernel(if_io_tqg) and speed is ~900Mbps.

[CJ]

Real life got in the way for a bit but I'm now back trying to get this working.

Both interfaces on the NUC are definitely set to 1Gbps.  I believe the brief speed test results of 1300 are a glitch caused by the way the sites are calculating the average.  It seems to be particularly common with fast.com.  The final result on these test is always ~900.

The network arrangement is:
Fibre -> ISP modem -> NUC -> Netgear switch -> PC

I normally run the Synology RT2600AC in place of the NUC, which is giving me close to line speed.  When the NUC is in place instead of the RT2600AC, the upload becomes slow.  Restarting the modem doesn't appear to make any difference.

During upload, top shows 20-30% on all 4 cores for kernel(if_io_tqg) and speed is ~150Mbps.
During download, I get ~10% on all cores for kernel(if_io_tqg) and speed is ~900Mbps.

How are you setting them to 1g?  Are the modem and switch ports 1g or something else?

As mentioned previously, what happens if you just use two local PCs with the NUC between them?  Try an iperf between the two.  You can also test iperf from your computer to the NUC.

I saw some commentors "fixing" their i225/i226 problems by putting a switch between the modem and their router.  Apparently some devices just don't agree with the cards.

One last thing you can try is loading Windows, Linux, and FreeBSD on the NUC and comparing the various iperf results to what you get with OPNSense.

[chamley]

How are you setting them to 1g?  Are the modem and switch ports 1g or something else?

As mentioned previously, what happens if you just use two local PCs with the NUC between them?  Try an iperf between the two.  You can also test iperf from your computer to the NUC.

I saw some commentors "fixing" their i225/i226 problems by putting a switch between the modem and their router.  Apparently some devices just don't agree with the cards.

One last thing you can try is loading Windows, Linux, and FreeBSD on the NUC and comparing the various iperf results to what you get with OPNSense.

Interfaces are set to 1000Base-T full-duplex in the Interface configuration menu in OPNsense.  The dashboard reports this as well.  All switch ports are configured to this as well.

iPerf to the NUC from the PC gives 0.96Gbps in both directions (10 streams).

Changed setup to:
PC -> Netgear switch -> NUC -> Other Netgear switch -> Laptop
(OPNsense is still doing NAT and firewall)
iPerf between PC and laptop gives 0.95Gbps in both directions (10 streams).

Back to the normal setup.
iPerf from the PC to a public server gives 0.2Gbps upload (10 streams).

Interesting idea of a switch between the router and modem.  I will try to find an unmanaged switch and test that.

[meyergru]

I have no experience with IPoE, but with PPPoE as an encapsulating protocol, there is an overhead for the data packets which often forces to have a smaller MTU set on the WAN interface. If that is not considered, connections can be much slower because of retries and/or refragmentation.

I.E.: You could try lowering the MTU of the WAN interface to somthing smaller like 1400 Bytes.