Hello all. I'm new to OPNsense and could use some help!
I have a 1000/1000 Mbps FTTH internet connection, but I'm struggling to get OPNsense to perform well on the uplink.
Previously I used a Synology RT2600AC as my router. With that, with both fast.com and speedtest.net I would get:
~ 800 Mbps down
~ 800 Mbps up
I've now switched to OPNsense 23.1.7 running on an Intel NUC with an i5-1340P, 16 GB RAM and dual Intel i226 NICs. I now get:
~ 1000 Mbps down. CPU utilisation is <4%
< 100 Mbps up. CPU utilisation is ~30%
Things I've checked/tried:
- Disabling spectre and meltdown mitigations.
- Disabling IPv6.
- WAN MTU. It's 1500, which I believe is correct for my provider.
- No intrusion detection or other packages active.
- Firewall is enabled (default configuration).
- VLANs. I am not using any and have not changed and VLAN-related settings.
- LAN DNS and DHCP. These are both disabled as they are provided by another device on the network.
- Hardware offloading is disabled for CRC, TSO and LRO.
That config should have no problem giving you line speed both ways, AFAIK. I don't have a connection to test that with, unfortunately.
What speeds were you getting with the default install? No changes to the config at all.
Have you tried putting the NUC between two machines locally and doing an iperf across them?
Hello! I'm sorry for replying so slowly. I missed the email notification!
I have just tried a completely default install of 22.7 and get:
~1000 Mbps down
~150 Mbps up
Back to the Synology router and I get:
~1000 Mbps down
~1000 Mbps up
Unfortunately I haven't had time to try iperf across the NUC.
I've just tried 23.1.9 with default setup and the results are the same, except CPU utilisation during upload is <10%.
Are you connecting to your ISP with PPPOE? You may want to set net.isr.dispatch to deferred if you do.
Additional to what @zan said. Try to change the tunnables. PPPoE behavies wierd.
OVerall your CPu should be able to do 1G on Single core without problem. However you are using a NUC, NUC usually has only a single onboard NIC.
Can you tell what NUC do you have?
How did you expand your NIC count?
What kind of adapter did you use?
Is that adapter okay?
Do you see any errors on the Interface that has the adapter?
Regards,
S.
Quote from: chamley on June 21, 2023, 10:56:44 PM
Hello! I'm sorry for replying so slowly. I missed the email notification!
I have just tried a completely default install of 22.7 and get:
~1000 Mbps down
~150 Mbps up
Back to the Synology router and I get:
~1000 Mbps down
~1000 Mbps up
Unfortunately I haven't had time to try iperf across the NUC.
No worries about the delay. I've been there before. How do you have the NUC connected? Is it set to 1g on each side? I know a lot of people have talked about the Intel 2.5g NICs having issues.
Also, are you actually seeing 1000 Mbps or are you rounding the 900ish gig limit? A lot of ISPs overprovision so if you're connected via 2.5g I would expect to see something like 1150 Mbps.
Quote from: Seimus on June 22, 2023, 10:02:17 AM
Additional to what @zan said. Try to change the tunnables. PPPoE behavies wierd.
OVerall your CPu should be able to do 1G on Single core without problem. However you are using a NUC, NUC usually has only a single onboard NIC.
Can you tell what NUC do you have?
How did you expand your NIC count?
What kind of adapter did you use?
Is that adapter okay?
Do you see any errors on the Interface that has the adapter?
Regards,
S.
I can't speak to the PPPoE but they stated in the OP that the NUC they're using has dual i226 2.5g NICs. Not all minis are single NIC machines.
Quote from: CJRoss on June 22, 2023, 03:01:59 PM
I can't speak to the PPPoE but they stated in the OP that the NUC they're using has dual i226 2.5g NICs. Not all minis are single NIC machines.
Yes i've seen, but its better to have it confirmed. Cause the mainstream NUC from Intel - Intel NUC 13 I am aware of usually had only 1 NIC.
Regards,
S.
Hello everyone, thanks for your replies. To answer your questions:
Are you connecting to your ISP with PPPOE? You may want to set net.isr.dispatch to deferred if you do. Yes, PPPoE. I tried a few tuneables yesterday: net.isr.dispatch=deferred, net.isr.bindthreads=1, net.isr.maxthreads=-1 This didn't make any difference.
EDIT: My mistake! It's not PPPoE. I have FTTP using IPoE.
Can you tell what NUC do you have? NUC13ANHI5
How did you expand your NIC count? This series of NUCs has a header for an additional NIC and 2x USB ports, same as previous generations of NUC that had two NICs. I am using the official Intel module. The USB ports are not in use.
What kind of adapter did you use? Intel i226
Is that adapter okay? Seems to work fine. I have tried swapping between the two NICs and they perform the same.
Do you see any errors on the Interface that has the adapter? No.
How do you have the NUC connected? Is it set to 1g on each side? Correct.
Also, are you actually seeing 1000 Mbps or are you rounding the 900ish gig limit? 900 is the average over most tests. For the first few seconds I can see up to ~1300. I guess the ISP is then throttling.
I've repeated more tests and the CPU usage is definitely spiking during upload.
I tried disabling the E-cores, leaving only the 4 P-cores. During upload, I hit ~150 Mbps with 50% CPU usage :o
Run top -PSH to show detailed CPU usage during your tests.
Quote from: chamley on June 22, 2023, 10:39:29 PM
How do you have the NUC connected? Is it set to 1g on each side? Correct.
Also, are you actually seeing 1000 Mbps or are you rounding the 900ish gig limit? 900 is the average over most tests. For the first few seconds I can see up to ~1300. I guess the ISP is then throttling.
If you're seeing 1300 then you don't have the NICs set to 1g. What does the Media say for each of the interfaces?
What devices do you have connected to the NUC? Please list everything from the fiber convertor/modem all the way to the PC you are testing from.
Real life got in the way for a bit but I'm now back trying to get this working.
Both interfaces on the NUC are definitely set to 1Gbps. I believe the brief speed test results of 1300 are a glitch caused by the way the sites are calculating the average. It seems to be particularly common with fast.com. The final result on these test is always ~900.
The network arrangement is:
Fibre -> ISP modem -> NUC -> Netgear switch -> PC
I normally run the Synology RT2600AC in place of the NUC, which is giving me close to line speed. When the NUC is in place instead of the RT2600AC, the upload becomes slow. Restarting the modem doesn't appear to make any difference.
During upload, top shows 20-30% on all 4 cores for kernel(if_io_tqg) and speed is ~150Mbps.
During download, I get ~10% on all cores for kernel(if_io_tqg) and speed is ~900Mbps.
Quote from: chamley on October 29, 2023, 03:21:46 PM
Real life got in the way for a bit but I'm now back trying to get this working.
Both interfaces on the NUC are definitely set to 1Gbps. I believe the brief speed test results of 1300 are a glitch caused by the way the sites are calculating the average. It seems to be particularly common with fast.com. The final result on these test is always ~900.
The network arrangement is:
Fibre -> ISP modem -> NUC -> Netgear switch -> PC
I normally run the Synology RT2600AC in place of the NUC, which is giving me close to line speed. When the NUC is in place instead of the RT2600AC, the upload becomes slow. Restarting the modem doesn't appear to make any difference.
During upload, top shows 20-30% on all 4 cores for kernel(if_io_tqg) and speed is ~150Mbps.
During download, I get ~10% on all cores for kernel(if_io_tqg) and speed is ~900Mbps.
How are you setting them to 1g? Are the modem and switch ports 1g or something else?
As mentioned previously, what happens if you just use two local PCs with the NUC between them? Try an iperf between the two. You can also test iperf from your computer to the NUC.
I saw some commentors "fixing" their i225/i226 problems by putting a switch between the modem and their router. Apparently some devices just don't agree with the cards.
One last thing you can try is loading Windows, Linux, and FreeBSD on the NUC and comparing the various iperf results to what you get with OPNSense.
Quote from: CJ on October 29, 2023, 03:30:58 PM
How are you setting them to 1g? Are the modem and switch ports 1g or something else?
As mentioned previously, what happens if you just use two local PCs with the NUC between them? Try an iperf between the two. You can also test iperf from your computer to the NUC.
I saw some commentors "fixing" their i225/i226 problems by putting a switch between the modem and their router. Apparently some devices just don't agree with the cards.
One last thing you can try is loading Windows, Linux, and FreeBSD on the NUC and comparing the various iperf results to what you get with OPNSense.
Interfaces are set to 1000Base-T full-duplex in the Interface configuration menu in OPNsense. The dashboard reports this as well. All switch ports are configured to this as well.
iPerf to the NUC from the PC gives 0.96Gbps in both directions (10 streams).
Changed setup to:
PC -> Netgear switch -> NUC -> Other Netgear switch -> Laptop
(OPNsense is still doing NAT and firewall)
iPerf between PC and laptop gives 0.95Gbps in both directions (10 streams).
Back to the normal setup.
iPerf from the PC to a public server gives 0.2Gbps upload (10 streams).
Interesting idea of a switch between the router and modem. I will try to find an unmanaged switch and test that.
I have no experience with IPoE, but with PPPoE as an encapsulating protocol, there is an overhead for the data packets which often forces to have a smaller MTU set on the WAN interface. If that is not considered, connections can be much slower because of retries and/or refragmentation.
I.E.: You could try lowering the MTU of the WAN interface to somthing smaller like 1400 Bytes.
Quote from: meyergru on October 29, 2023, 08:18:25 PM
I have no experience with IPoE, but with PPPoE as an encapsulating protocol, there is an overhead for the data packets which often forces to have a smaller MTU set on the WAN interface. If that is not considered, connections can be much slower because of retries and/or refragmentation.
I.E.: You could try lowering the MTU of the WAN interface to somthing smaller like 1400 Bytes.
I wondered about MTU as well. 1500 appears to be the correct value for my connection. This is what the Synology router uses, and testing with different ping packet sizes indicates that 1500 is ok. Thank you for the suggestion!
I'm worried that searches for "OPNsense slow upload" or "pfsense slow upload" show lots of forum/Reddit posts with problems similar to mine, and the conclusion is often that something changed in freeBSD which is the cause :(
Quote from: chamley on October 29, 2023, 11:51:19 PM
Quote from: meyergru on October 29, 2023, 08:18:25 PM
I have no experience with IPoE, but with PPPoE as an encapsulating protocol, there is an overhead for the data packets which often forces to have a smaller MTU set on the WAN interface. If that is not considered, connections can be much slower because of retries and/or refragmentation.
I.E.: You could try lowering the MTU of the WAN interface to somthing smaller like 1400 Bytes.
I wondered about MTU as well. 1500 appears to be the correct value for my connection. This is what the Synology router uses, and testing with different ping packet sizes indicates that 1500 is ok. Thank you for the suggestion!
I'm worried that searches for "OPNsense slow upload" or "pfsense slow upload" show lots of forum/Reddit posts with problems similar to mine, and the conclusion is often that something changed in freeBSD which is the cause :(
When you tested MTU with ping did you set the DF flag on? This will tel lyou what is the highest non Fragmented MTU size thru your provider. Without setting the DF bit you can ping any MTU size because any L3 HOP can de-facto fragment the packet to the needed MTU size they have set on their egress interface.
Regards,
S.
Quote from: chamley on October 29, 2023, 05:50:04 PM
Interfaces are set to 1000Base-T full-duplex in the Interface configuration menu in OPNsense. The dashboard reports this as well. All switch ports are configured to this as well.
iPerf to the NUC from the PC gives 0.96Gbps in both directions (10 streams).
Changed setup to:
PC -> Netgear switch -> NUC -> Other Netgear switch -> Laptop
(OPNsense is still doing NAT and firewall)
iPerf between PC and laptop gives 0.95Gbps in both directions (10 streams).
Back to the normal setup.
iPerf from the PC to a public server gives 0.2Gbps upload (10 streams).
Interesting idea of a switch between the router and modem. I will try to find an unmanaged switch and test that.
It does seem to imply that the problem is with the NUC and your modem. Another test you can try is to put the old router in place and then the NUC behind it and test that. Additionally, hang a laptop off of the other router as well so you're testing against a local network.
I'm not sure if the type of switch matters but then I don't know if the switch will fix your issue or not. It's just something that I came across when I was researching the i225/i226 NICs. From what I've been able to gather, the issues are primarily in the embedded versions. I don't think I saw anyone posting about problems with PCIe versions. I'm currently using an official Intel i225 PCIe NIC and prior to that an off brand one and both have been rock solid.
Quote from: Seimus on October 30, 2023, 10:24:38 AM
When you tested MTU with ping did you set the DF flag on? This will tel lyou what is the highest non Fragmented MTU size thru your provider. Without setting the DF bit you can ping any MTU size because any L3 HOP can de-facto fragment the packet to the needed MTU size they have set on their egress interface.
Regards,
S.
Yes I did :)
Quote from: CJ on October 30, 2023, 01:36:39 PM
It does seem to imply that the problem is with the NUC and your modem. Another test you can try is to put the old router in place and then the NUC behind it and test that. Additionally, hang a laptop off of the other router as well so you're testing against a local network.
I'm not sure if the type of switch matters but then I don't know if the switch will fix your issue or not. It's just something that I came across when I was researching the i225/i226 NICs. From what I've been able to gather, the issues are primarily in the embedded versions. I don't think I saw anyone posting about problems with PCIe versions. I'm currently using an official Intel i225 PCIe NIC and prior to that an off brand one and both have been rock solid.
Thanks CJ. I'll try doing some more tests and post the results.
I agree, it does seem like it's between the NUC and modem. When the connection is being established, does the router set any parameters? Could it be somehow misconfigurating the connection?
Its negotiates speed and duplex if its set for auto. There could be a slight possibility of speed/duplex mismatch on the modem side. On OPN you can clearly see whats is set, can you see it or log into to modem as well to check it?
Regards,
S.
Quote from: chamley on October 30, 2023, 09:46:28 PM
Thanks CJ. I'll try doing some more tests and post the results.
I agree, it does seem like it's between the NUC and modem. When the connection is being established, does the router set any parameters? Could it be somehow misconfigurating the connection?
No. My guess is that the problem is with your i226 NICs. No idea why you would see slow upload speeds, though. I would have expected the problems to be both ways.
Quote from: CJ on October 31, 2023, 01:26:35 PM
Quote from: chamley on October 30, 2023, 09:46:28 PM
Thanks CJ. I'll try doing some more tests and post the results.
I agree, it does seem like it's between the NUC and modem. When the connection is being established, does the router set any parameters? Could it be somehow misconfigurating the connection?
No. My guess is that the problem is with your i226 NICs. No idea why you would see slow upload speeds, though. I would have expected the problems to be both ways.
Ok, I've now tested with an unmanaged gigabit switch between the modem and OPNsense. Speedtest results are the same, and OPNsense still has high CPU usage during the upload test. As before, top shows that if_io_tqg is taking the CPU resources.
Quote from: Seimus on October 31, 2023, 10:19:59 AM
Its negotiates speed and duplex if its set for auto. There could be a slight possibility of speed/duplex mismatch on the modem side. On OPN you can clearly see whats is set, can you see it or log into to modem as well to check it?
Regards,
S.
OPNsense has both WAN and LAN set to 1000baseT full-duplex. I can't login to the modem.
Quote from: chamley on November 13, 2023, 06:52:29 PM
Ok, I've now tested with an unmanaged gigabit switch between the modem and OPNsense. Speedtest results are the same, and OPNsense still has high CPU usage during the upload test. As before, top shows that if_io_tqg is taking the CPU resources.
I missed that bit earlier. It looks like it's not an i226 issue. This thread may have some useful information, but it appears to be an issue that has affected a variety of different configurations.
https://forum.opnsense.org/index.php?topic=18754.30
Have you tried testing with just the default install config? And are you still using 23.1 or have you switched to 23.7? Might be worth a reinstall and testing with no changes.
It may also be worth trying this type from the above linked thread.
https://forum.opnsense.org/index.php?topic=18754.msg159739#msg159739
Quote from: CJ on November 14, 2023, 05:51:54 PM
I missed that bit earlier. It looks like it's not an i226 issue. This thread may have some useful information, but it appears to be an issue that has affected a variety of different configurations.
https://forum.opnsense.org/index.php?topic=18754.30
Have you tried testing with just the default install config? And are you still using 23.1 or have you switched to 23.7? Might be worth a reinstall and testing with no changes.
It may also be worth trying this type from the above linked thread.
https://forum.opnsense.org/index.php?topic=18754.msg159739#msg159739
Thanks, that looks like a similar problem. There are some posts elsewhere that describe a similar problem too.
I'm still using 23.1 but I will test 23.7 and let you know the result. I've tested pfSense CE 2.7.0 and that has the exact same problem. I was hoping that FreeBSD 14 might fix this from 13.1, but I guess not.
I have been using the default configuration of 23.1. Then I've tried a couple of tunables:
net.isr.dispatch=deferred
net.isr.maxthreads=-1
net.isr.bindthreads = 1 # This combination appeared to give a small improvement
hw.ibrs_disable=1 # Also appeared to give a small improvement
Enabled powerd and set to Maximum # No clear improvement
That gets me to 200 Mbps upload. Better, but still a way to go.
What are your clock speeds showing? Is the CPU boosting up to 4.6? Seems odd that you'd have issues with that fast a chip.