OPNsense Forum
English Forums => Hardware and Performance => Topic started by: Ric878 on August 05, 2018, 03:35:46 am
-
Hi,
I recently upgraded to Gig internet with Comcast and have been having trouble getting my full speed when traffic is going through the firewall.
To do some baseline testing, I have connected my laptop directly to the cable modem and when I do a speed test on http://speedtest.xfinity.com/ I consistently get between 900 - 940 Mbps. When I do the same speed test with OPNsense as the firelwall/router, my download speeds are ~ 600 Mbps. I have an IPSec vpn running but I have removed it and run the speed tests and still see the same results. For testing pruposes, I have all other devices removed from the switch so it's just my laptop, the OPNsense box, and the cable modem. The laptop is connected via ethernet cable during the tests. If I remove the OPNsense box from the equation, I consistently get the faster 900 - 940 Mbps speed test results.
I have done an iperf3 test from my laptop to the OPNsense LAN port and am getting ~940 Mbps test results, that of course is not traversing the firewall.
Here are the OPNsense box specs:
Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz (4 cores)
8 Gigs of Ram
I350 Quad Port Network Card
Modem:
Motorola MB8600
Any troubleshooting tips would be appreciated.
Thanks,
Ric
-
Set MSS on LAN to 1400
-
Set MSS on LAN to 1400
Tried that, no change. Thanks for the suggestion though.
-
IPS and Proxy disabled?
-
IPS and Proxy disabled?
Yes, both are disabled.
-
If I remove the OPNsense box from the equation, I consistently get the faster 900 - 940 Mbps speed test results.
When you add the OPNsense box, is the modem/connection interface bridged properly? No double NATing or other double-ups?
-
If I remove the OPNsense box from the equation, I consistently get the faster 900 - 940 Mbps speed test results.
When you add the OPNsense box, is the modem/connection interface bridged properly? No double NATing or other double-ups?
No double NATing. As a matter of fact, the modem doesn't have any router or firewall features, other than passing a local IP during boot before it passes a public IP. For reference, it is a Motoroal MB8600.
Thanks,
Ric
-
Just wanted to update with some additional steps I've taken without success. So far I've tried to enable "hardware checksum offload", and "hardware TCP segmentation offload", rebooted and still had no improvement. I left "hardware large receive offload" disabled during testing.
I have also replaced all Ethernet cables just to be sure that wasn't part of the issue.
-
Perhaps check out https://www.freebsd.org/cgi/man.cgi?tuning
dcol also has a more specific tuning thread here. I found it helpful: https://forum.opnsense.org/index.php?topic=6590.0
-
Perhaps check out https://www.freebsd.org/cgi/man.cgi?tuning
dcol also has a more specific tuning thread here. I found it helpful: https://forum.opnsense.org/index.php?topic=6590.0
@youngman Thanks for those links. I'm playing around with some of the tunables now. I'll report back if there's any improvement.
-
@Ric878, you and I are in the same boat brother. I have not found a solution yet, but been tinkering for a few weeks.
To rule out routing, IPS, etc...I have been testing with a python script so only the WAN interface is in scope. If I plug my MacBook into the MB8600 I get between 930-950. With my OPNsense box I see a max of 600-630.
This test shows only 428.87, but its the middle of the day ;-)
compunction@opnsense:~ % ./speedtest-cli.py
Retrieving speedtest.net configuration...
Testing from Comcast Cable (73.237.238.141)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Comcast (Atlanta, GA) [20.61 km]: 12.635 ms
Testing download speed................................................................................
Download: 428.87 Mbit/s
Testing upload speed................................................................................................
Upload: 41.34 Mbit/s
compunction@opnsense:~ %
A few things I have tried:
net.inet.tcp.tso="0"
dev.igb.0.fc="0"
dev.em.0.fc="0"
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
hw.em.rx_process_limit="-1"
hw.em.tx_process_limit="-1"
dev.igb.0.eee_disabled="1"
machdep.hyperthreading_allowed="0"
hw.igb.txd="2048"
hw.igb.rxd="2048"
hw.em.txd="2048"
hw.em.rxd="2048"
net.link.ifqmaxlen="4096"
hw.igb.max_interrupt_rate="32000"
hw.em.max_interrupt_rate="32000"
net.inet.tcp.sendbuf_inc="32768"
net.inet.tcp.recvbuf_inc="32768"
net.inet.tcp.sendbuf_max="4194304"
net.inet.tcp.recvbuf_max="4194304"
I have also tried swapping cat5e cables...no luck.
Still in the search for a resolution :).
-
@compunction, I'm still playing with different combinations of settings with mixed results. Sometimes, I feel like I find a combination that seems to improve things, but the results don't seem to last. So, I am starting to think that my test methodology is flawed.
First, using speed test sites seems to be very inconsistent. I'm thinking that the only real way to get an accurate speed test is to do an iperf test with two computers, on either side of the OPNSense router (LAN/WAN). To be honest, I feel I may even be getting throttled from hitting the speed test sites so often. Just a paranoid theory really.
I'll report back once I get more info and setup a proper test. That said, what are the specs of your OPNSense box?
-
Intel(R) Core(TM) i7-8809G CPU @ 3.10GHz (3096.15-MHz K8-class CPU)
real memory = 34359738368 (32768 MB)
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k>
em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k>
I have wondered myself if they could be throttling me, but when I test speed test on two different sites from two different machines, I see the speed drop as I would expect. But general internet latency is likely a factor as well. I need to grab my old hardware and setup a iperf box on the wan side. I could likely put a 192.168.100.x IP on it has I have a route for that to get the router interface. (since I only have one WAN IP).
-
Okay, so I did some new tests with iperf3 but I need someone to validate the setup and let me know if this is a valid test.
My OPNSense box has a 4 port network card in it. What I did was create a VLAN on my switch (VLAN 10), and created a new VLAN interface in OPNSense and attached it to one of my unused ports. I then connected this port to a switch port that was tagged for VLAN 10.
I also setup a spare PC with Debian 8 (PC2) on it and connected it to a static port on the switch for VLAN 10. I verified that without the OPNSense box in the middle, that I could not access the PC2 on VLAN10 from my test PC (PC1) on the default VLAN.
I then added the proper firewall rules on OPNSense to allow access to PC2 from PC1. All is good at this point. I proceeded to run my iperf3 test with PC1 as the client and PC2 as the server. My results were fantastic, ~915 Mbits/s
This is where I need someone to validate my test, my thinking is that because PC2 is on a separate VLAN that my OPNSense box had to route and allow all the iperf traffic through the firewall, proving that the OPNSense box was very capable of near 1 Gbs traffic. Question is, was my test valid. Only thing I can think about is that there was no NAT involved in the test. Would that make a significant enough difference in this validation?
-
Why not enabling NAT for testing?
Firewall : NAT : Outbound
Set to manual or hybrid, then add a rule, source PC1 LAN, dest PC2 LAN, translated address PC2 address.
-
Actually, I just found some public iperf3 servers on the internet and was able to run a few tests and can verify that I am able to route traffic through the OPNSense box at near Gig speeds (~915 Mbits/sec). As far as I can tell, this verifies that the OPNSense box is running okay.
What does this all mean as far as internet speed tests are concerned? I'm not exactly sure. I have read over and over again that they are inconsistent and not a great measure of a speed test but I can't stop thinking about the fact that if I run the test directly from my laptop to the cable modem that I get better results. At this point, maybe I'm just chasing ghosts and should be satisfied with the results I'm getting with iperf.
-
You should be satisfied! 8)
Public iperf servers don't offer equal results.
My lab consists of 4 systems, two firewalls and two clients behind, so always get good results.
But also iperf doesn't always deliver consistent results itself. And also keep in mind that iperf3 is not multithreaded. You should use iperf2.
Also it always depends on the use case .. some guys want to get 1 GB on 1 stream which is hard to achieve on FreeBSD. If you run multiple streams it's really easy to achieve higher rates. In all my use cases I never need only ONE single GB stream, so I'm always fine with OPN performance :)
-
Also it always depends on the use case .. some guys want to get 1 GB on 1 stream which is hard to achieve on FreeBSD. If you run multiple streams it's really easy to achieve higher rates. In all my use cases I never need only ONE single GB stream, so I'm always fine with OPN performance :)
That is some very interesting info! I've just now realised that the sites I used to test when setting things up allowed multithread testing e.g. https://testmy.net/ Maybe that is why they always seemed to give me better numbers?!
That said, now that OP has ruled out any fundamental setup issues and has achieved raw speed, IMO the best thing to start chasing now is lower buffer bloat :devil: - check out for example http://www.dslreports.com/speedtest
Good luck!
-
Interesting, so single TCP session vs multiple TCP sessions.
Did a quick and dirty test form my OPNsense box just to test the WAN. Please do not make fun of my scripting, I could have done a for loop and done it 100 different ways, but I am not a dev :)
The output doing a speedtest to 12 different servers are the same time yielded 932.92Mbps. I would say that is close enough and does prove that the bandwidth is there, just not for a single tcp session.
----------SCRIPT----------
#!/bin/tcsh
/home/compunction/speedtest-cli.py --server 11143 > test.1 &
/home/compunction/speedtest-cli.py --server 1767 > test.2 &
/home/compunction/speedtest-cli.py --server 10391 > test.3 &
/home/compunction/speedtest-cli.py --server 8169 > test.4 &
/home/compunction/speedtest-cli.py --server 13653 > test.5 &
/home/compunction/speedtest-cli.py --server 13655 > test.6 &
/home/compunction/speedtest-cli.py --server 10575 > test.7 &
/home/compunction/speedtest-cli.py --server 3595 > test.8 &
/home/compunction/speedtest-cli.py --server 5296 > test.9 &
/home/compunction/speedtest-cli.py --server 8707 > test.10 &
/home/compunction/speedtest-cli.py --server 12407 > test.11 &
/home/compunction/speedtest-cli.py --server 15853 > test.12 &
sleep 30
/bin/cat /home/compunction/test.* | /usr/bin/grep Download | /usr/bin/awk '{print $2}' > test.output
/bin/cat /home/compunction/test.output
echo "Total: "
/bin/cat /home/compunction/test.output | /usr/bin/awk '{sum+=$1} END{print sum}'
/bin/rm /home/compunction/test.*
----------OUTPUT----------
40.09
42.32
69.85
81.78
288.80
41.57
44.78
45.37
86.77
50.05
64.90
76.64
Total:
932.92
-
Yes, I can not let it go...I have still been tinkering with this when time permits.
I found something interesting that maybe an issue with comcast. Out-of-Order packets!
From netstat -s
43819 out-of-order packets (63444003 bytes)
18910 discarded due to memory problems
These counters only increment when I do a speedtest.net.
Setting this net.inet.tcp.reass.maxqueuelen="1000"
The default is 100
I was able to prevent out-of-oder packets from being dropped. As I understand it this limit was put in due to a security vulnerability involving packet reassembly. It appears Comcast has an issue with out-of-order packets so I had to give it a little more room to work.
Unfortunately it did not help my speed (likely hitting a different limit).
I suspect this would not have turned up in an iperf as packets are likely not coming in out-of-order.
-
Sorry to necro this post but I didn't see anyone suggest manually setting the interfaces to 1000 BaseT full duplex. I had a similar issue where I was being limited to about 100Mbps. Turns out, OPNsense (or one of my other devices) was auto negotiating a lower speed. Setting it manually instantly fixed my issue.
-
Set MSS on LAN to 1400
Yes, that did the trick for me! Thanks!!
-
Sorry to resurrect this older thread but I'm on my 2nd week trying to get my customers OPNSENSE firewall to increase throughput. At the present time i have OPNSENSE running as a VM under Proxmox. All network resources are being recognized as 10GBE. We have increased our ISP account up to 1 Gbps.
Just like the previous poster if i plug my laptop into the modem i get speeds approaching 1 Gbps. When i run through OPNSENSE and plug directly into the LAN port it appears that OPNSENSE throttles the speeds to no more than 360 Gbps.
I've edited the LAN MSS with 1400 as recommended....but no changes.
Nothing special with OPNSENSE - Just a fresh install with the WAN and LAN ports. No NAT, proxies, etc.
In testing I've tired a few other firewall packages (I hate them) but wanted to see if i was dealing with a hardware issue. I've installed PFSENSE and have the same issues....throughput is throttled to about 360 Gbps.
However, i've installed IPFIRE - using the exact same hardware configuration and SURPRISE! I'm able to achieve almost 1 Gbps throughput.
I'm confident this is some type of configuration setting......I love OPNSENSE but need to get this figured out.
Any help would be appreciated!!
-
Sounds like Options of the VM. Real Hardware doesnt have this problem
-
Just like the previous poster if i plug my laptop into the modem i get speeds approaching 1 Gbps. When i run through OPNSENSE and plug directly into the LAN port it appears that OPNSENSE throttles the speeds to no more than 360 Gbps.
I've edited the LAN MSS with 1400 as recommended....but no changes.
Nothing special with OPNSENSE - Just a fresh install with the WAN and LAN ports. No NAT, proxies, etc.
In testing I've tired a few other firewall packages (I hate them) but wanted to see if i was dealing with a hardware issue. I've installed PFSENSE and have the same issues....throughput is throttled to about 360 Gbps.
However, i've installed IPFIRE - using the exact same hardware configuration and SURPRISE! I'm able to achieve almost 1 Gbps throughput.
I'm confident this is some type of configuration setting......I love OPNSENSE but need to get this figured out.
Any help would be appreciated!!
I've had similar results. I"m using AT&T 1gb Fiber, and recently switched from Untangle to OPNsense. With Untangle I was regularly getting ~700 down, and ~900 up. With OPNsense I'm getting ~350 down, and ~200 up.
I've tried clamping MSS to 1400, which seemed to slow it down slightly. Then as someone else suggested I did a test with multiple streams, setting both upload and download to 8 streams, which was even worse at ~130 down and ~140 up (combined).
I"m not really sure where to go from here.
-
I too am experiencing this issue in Australia with an NBN HFC connection. I have a commercial firewall which i get > 900mbps but down to under 300 with opnsense.I have tried the MSS setting which slows it down slightly.
Its a real shame as it makes the solution unusable for me, if anyone has an ideas I'm open otherwise I guess its ipfire for me.
Cheers
-
https://docs.opnsense.org/troubleshooting/hardening.html
Did you check meltdown stuff?