Messages

I think I might have the same problem as you:
https://forum.opnsense.org/index.php?topic=33878.0

I have not found a solution yet  :(

I missed that bit earlier.  It looks like it's not an i226 issue.  This thread may have some useful information, but it appears to be an issue that has affected a variety of different configurations.

https://forum.opnsense.org/index.php?topic=18754.30

Have you tried testing with just the default install config?  And are you still using 23.1 or have you switched to 23.7?  Might be worth a reinstall and testing with no changes.

It may also be worth trying this type from the above linked thread.

https://forum.opnsense.org/index.php?topic=18754.msg159739#msg159739

Thanks, that looks like a similar problem.  There are some posts elsewhere that describe a similar problem too.

I'm still using 23.1 but I will test 23.7 and let you know the result.  I've tested pfSense CE 2.7.0 and that has the exact same problem.  I was hoping that FreeBSD 14 might fix this from 13.1, but I guess not.

I have been using the default configuration of 23.1.  Then I've tried a couple of tunables:

net.isr.dispatch=deferred
net.isr.maxthreads=-1
net.isr.bindthreads = 1  # This combination appeared to give a small improvement

hw.ibrs_disable=1  # Also appeared to give a small improvement

Enabled powerd and set to Maximum  # No clear improvement

That gets me to 200 Mbps upload.  Better, but still a way to go. 

Thanks CJ. I'll try doing some more tests and post the results.
I agree, it does seem like it's between the NUC and modem. When the connection is being established, does the router set any parameters? Could it be somehow misconfigurating the connection?

No. My guess is that the problem is with your i226 NICs.  No idea why you would see slow upload speeds, though.  I would have expected the problems to be both ways.

Ok, I've now tested with an unmanaged gigabit switch between the modem and OPNsense.  Speedtest results are the same, and OPNsense still has high CPU usage during the upload test.  As before, top shows that if_io_tqg is taking the CPU resources.

Its negotiates speed and duplex if its set for auto. There could be a slight possibility of speed/duplex mismatch on the modem side. On OPN you can clearly see whats is set, can you see it or log into to modem as well to check it?

Regards,
S.

OPNsense has both WAN and LAN set to 1000baseT full-duplex.  I can't login to the modem.

When you tested MTU with ping did you set the DF flag on? This will tel lyou what is the highest non Fragmented MTU size thru your provider. Without setting the DF bit you can ping any MTU size because any L3 HOP can de-facto fragment the packet to the needed MTU size they have set on their egress interface.

Regards,
S.

Yes I did  :)

It does seem to imply that the problem is with the NUC and your modem.  Another test you can try is to put the old router in place and then the NUC behind it and test that.  Additionally, hang a laptop off of the other router as well so you're testing against a local network.

I'm not sure if the type of switch matters but then I don't know if the switch will fix your issue or not.  It's just something that I came across when I was researching the i225/i226 NICs.  From what I've been able to gather, the issues are primarily in the embedded versions.  I don't think I saw anyone posting about problems with PCIe versions.  I'm currently using an official Intel i225 PCIe NIC and prior to that an off brand one and both have been rock solid.

Thanks CJ. I'll try doing some more tests and post the results.
I agree, it does seem like it's between the NUC and modem. When the connection is being established, does the router set any parameters? Could it be somehow misconfigurating the connection?

I have no experience with IPoE, but with PPPoE as an encapsulating protocol, there is an overhead for the data packets which often forces to have a smaller MTU set on the WAN interface. If that is not considered, connections can be much slower because of retries and/or refragmentation.

I.E.: You could try lowering the MTU of the WAN interface to somthing smaller like 1400 Bytes.

I wondered about MTU as well. 1500 appears to be the correct value for my connection. This is what the Synology router uses, and testing with different ping packet sizes indicates that 1500 is ok. Thank you for the suggestion!

I'm worried that searches for "OPNsense slow upload" or "pfsense slow upload" show lots of forum/Reddit posts with problems similar to mine, and the conclusion is often that something changed in freeBSD which is the cause :(

How are you setting them to 1g?  Are the modem and switch ports 1g or something else?

As mentioned previously, what happens if you just use two local PCs with the NUC between them?  Try an iperf between the two.  You can also test iperf from your computer to the NUC.

I saw some commentors "fixing" their i225/i226 problems by putting a switch between the modem and their router.  Apparently some devices just don't agree with the cards.

One last thing you can try is loading Windows, Linux, and FreeBSD on the NUC and comparing the various iperf results to what you get with OPNSense.

Interfaces are set to 1000Base-T full-duplex in the Interface configuration menu in OPNsense.  The dashboard reports this as well.  All switch ports are configured to this as well.

iPerf to the NUC from the PC gives 0.96Gbps in both directions (10 streams).

Changed setup to:
PC -> Netgear switch -> NUC -> Other Netgear switch -> Laptop
(OPNsense is still doing NAT and firewall)
iPerf between PC and laptop gives 0.95Gbps in both directions (10 streams).

Back to the normal setup.
iPerf from the PC to a public server gives 0.2Gbps upload (10 streams).

Interesting idea of a switch between the router and modem.  I will try to find an unmanaged switch and test that.

Real life got in the way for a bit but I'm now back trying to get this working.

Both interfaces on the NUC are definitely set to 1Gbps.  I believe the brief speed test results of 1300 are a glitch caused by the way the sites are calculating the average.  It seems to be particularly common with fast.com.  The final result on these test is always ~900.

The network arrangement is:
Fibre -> ISP modem -> NUC -> Netgear switch -> PC

I normally run the Synology RT2600AC in place of the NUC, which is giving me close to line speed.  When the NUC is in place instead of the RT2600AC, the upload becomes slow.  Restarting the modem doesn't appear to make any difference.

During upload, top shows 20-30% on all 4 cores for kernel(if_io_tqg) and speed is ~150Mbps.
During download, I get ~10% on all cores for kernel(if_io_tqg) and speed is ~900Mbps.

[Yes, PPPoE. I tried a few tuneables yesterday: net.isr.dispatch=deferred, net.isr.bindthreads=1, net.isr.maxthreads=-1 This didn't make any difference.]

Hello everyone, thanks for your replies.  To answer your questions:

Are you connecting to your ISP with PPPOE? You may want to set net.isr.dispatch to deferred if you do.  Yes, PPPoE.  I tried a few tuneables yesterday:  net.isr.dispatch=deferred, net.isr.bindthreads=1, net.isr.maxthreads=-1  This didn't make any difference.
EDIT: My mistake! It's not PPPoE. I have FTTP using IPoE.

Can you tell what NUC do you have?  NUC13ANHI5
How did you expand your NIC count?  This series of NUCs has a header for an additional NIC and 2x USB ports, same as previous generations of NUC that had two NICs.  I am using the official Intel module.  The USB ports are not in use.
What kind of adapter did you use?  Intel i226
Is that adapter okay?  Seems to work fine.  I have tried swapping between the two NICs and they perform the same.
Do you see any errors on the Interface that has the adapter?  No.

How do you have the NUC connected?  Is it set to 1g on each side?  Correct.
Also, are you actually seeing 1000 Mbps or are you rounding the 900ish gig limit?  900 is the average over most tests.  For the first few seconds I can see up to ~1300.  I guess the ISP is then throttling.


I've repeated more tests and the CPU usage is definitely spiking during upload.
I tried disabling the E-cores, leaving only the 4 P-cores.  During upload, I hit ~150 Mbps with 50% CPU usage  :o

I've just tried 23.1.9 with default setup and the results are the same, except CPU utilisation during upload is <10%.

Hello!  I'm sorry for replying so slowly.  I missed the email notification!

I have just tried a completely default install of 22.7 and get:
~1000 Mbps down
~150 Mbps up

Back to the Synology router and I get:
~1000 Mbps down
~1000 Mbps up

Unfortunately I haven't had time to try iperf across the NUC.

Did you fix this?  I have a similar problem with upload being slower than my previous router on a symmetric connection.

Did you fix this?  I have a similar problem.

[< 100 Mbps up.]

Hello all.  I'm new to OPNsense and could use some help!

I have a 1000/1000 Mbps FTTH internet connection, but I'm struggling to get OPNsense to perform well on the uplink.

Previously I used a Synology RT2600AC as my router.  With that, with both fast.com and speedtest.net I would get:
~ 800 Mbps down
~ 800 Mbps up

I've now switched to OPNsense 23.1.7 running on an Intel NUC with an i5-1340P, 16 GB RAM and dual Intel i226 NICs.  I now get:
~ 1000 Mbps down.  CPU utilisation is <4%
< 100 Mbps up.  CPU utilisation is ~30%

Things I've checked/tried:
- Disabling spectre and meltdown mitigations.
- Disabling IPv6.
- WAN MTU.  It's 1500, which I believe is correct for my provider.
- No intrusion detection or other packages active.
- Firewall is enabled (default configuration).
- VLANs.  I am not using any and have not changed and VLAN-related settings.
- LAN DNS and DHCP.  These are both disabled as they are provided by another device on the network.
- Hardware offloading is disabled for CRC, TSO and LRO.