Bandwidth slow through firewall

[Ric878]

Actually, I just found some public iperf3 servers on the internet and was able to run a few tests and can verify that I am able to route traffic through the OPNSense box at near Gig speeds (~915 Mbits/sec). As far as I can tell, this verifies that the OPNSense box is running okay.

What does this all mean as far as internet speed tests are concerned? I'm not exactly sure. I have read over and over again that they are inconsistent and not a great measure of a speed test but I can't stop thinking about the fact that if I run the test directly from my laptop to the cable modem that I get better results. At this point, maybe I'm just chasing ghosts and should be satisfied with the results I'm getting with iperf.

[mimugmail]

You should be satisfied! 

Public iperf servers don't offer equal results.
My lab consists of 4 systems, two firewalls and two clients behind, so always get good results.

But also iperf doesn't always deliver consistent results itself. And also keep in mind that iperf3 is not multithreaded. You should use iperf2.

Also it always depends on the use case .. some guys want to get 1 GB on 1 stream which is hard to achieve on FreeBSD. If you run multiple streams it's really easy to achieve higher rates. In all my use cases I never need only ONE single GB stream, so I'm always fine with OPN performance

[youngman]

Also it always depends on the use case .. some guys want to get 1 GB on 1 stream which is hard to achieve on FreeBSD. If you run multiple streams it's really easy to achieve higher rates. In all my use cases I never need only ONE single GB stream, so I'm always fine with OPN performance

That is some very interesting info! I've just now realised that the sites I used to test when setting things up allowed multithread testing e.g. https://testmy.net/ Maybe that is why they always seemed to give me better numbers?!

That said, now that OP has ruled out any fundamental setup issues and has achieved raw speed, IMO the best thing to start chasing now is lower buffer bloat :devil: - check out for example http://www.dslreports.com/speedtest

Good luck!

[compunction]

Interesting, so single TCP session vs multiple TCP sessions.

Did a quick and dirty test form my OPNsense box just to test the WAN.  Please do not make fun of my scripting, I could have done a for loop and done it 100 different ways, but I am not a dev

The output doing a speedtest to 12 different servers are the same time yielded 932.92Mbps.  I would say that is close enough and does prove that the bandwidth is there, just not for a single tcp session.

----------SCRIPT----------
#!/bin/tcsh
/home/compunction/speedtest-cli.py --server 11143 > test.1 &
/home/compunction/speedtest-cli.py --server 1767 > test.2 &
/home/compunction/speedtest-cli.py --server 10391 > test.3 &
/home/compunction/speedtest-cli.py --server 8169 > test.4 &
/home/compunction/speedtest-cli.py --server 13653 > test.5 &
/home/compunction/speedtest-cli.py --server 13655 > test.6 &
/home/compunction/speedtest-cli.py --server 10575 > test.7 &
/home/compunction/speedtest-cli.py --server 3595 > test.8 &
/home/compunction/speedtest-cli.py --server 5296 > test.9 &
/home/compunction/speedtest-cli.py --server 8707 > test.10 &
/home/compunction/speedtest-cli.py --server 12407 > test.11 &
/home/compunction/speedtest-cli.py --server 15853 > test.12 &
sleep 30
/bin/cat /home/compunction/test.* | /usr/bin/grep Download | /usr/bin/awk '{print $2}' > test.output
/bin/cat /home/compunction/test.output
echo "Total: "
/bin/cat /home/compunction/test.output | /usr/bin/awk '{sum+=$1} END{print sum}'
/bin/rm /home/compunction/test.*

----------OUTPUT----------
40.09
42.32
69.85
81.78
288.80
41.57
44.78
45.37
86.77
50.05
64.90
76.64
Total:
932.92

[compunction]

Yes, I can not let it go...I have still been tinkering with this when time permits.

I found something interesting that maybe an issue with comcast.  Out-of-Order packets!

From netstat -s

43819 out-of-order packets (63444003 bytes)
18910 discarded due to memory problems

These counters only increment when I do a speedtest.net.

Setting this net.inet.tcp.reass.maxqueuelen="1000"
The default is 100

I was able to prevent out-of-oder packets from being dropped.  As I understand it this limit was put in due to a security vulnerability involving packet reassembly.  It appears Comcast has an issue with out-of-order packets so I had to give it a little more room to work.

Unfortunately it did not help my speed (likely hitting a different limit).

I suspect this would not have turned up in an iperf as packets are likely not coming in out-of-order.

[samsonmcnulty]

Sorry to necro this post but I didn't see anyone suggest manually setting the interfaces to 1000 BaseT full duplex. I had a similar issue where I was being limited to about 100Mbps. Turns out, OPNsense (or one of my other devices) was auto negotiating a lower speed. Setting it manually instantly fixed my issue.

[yapsr]

Set MSS on LAN to 1400

Yes, that did the trick for me! Thanks!!

[rocketlaywer]

Sorry to resurrect this older thread but I'm on my 2nd week trying to get my customers OPNSENSE firewall to increase throughput.  At the present time i have OPNSENSE running as a VM under Proxmox.  All network resources are being recognized as 10GBE.  We have increased our ISP account up to 1 Gbps.

Just like the previous poster if i plug my laptop into the modem i get speeds approaching 1 Gbps.  When i run through OPNSENSE and plug directly into the LAN port it appears that OPNSENSE throttles the speeds to no more than 360 Gbps.

I've edited the LAN MSS with 1400 as recommended....but no changes.

Nothing special with OPNSENSE - Just a fresh install with the WAN and LAN ports.  No NAT, proxies, etc.

In testing I've tired a few other firewall packages (I hate them) but wanted to see if i was dealing with a hardware issue.  I've installed PFSENSE and have the same issues....throughput is throttled to about 360 Gbps.

However, i've installed IPFIRE - using the exact same hardware configuration and SURPRISE!  I'm able to achieve almost 1 Gbps throughput.

I'm confident this is some type of configuration setting......I love OPNSENSE but need to get this figured out.

Any help would be appreciated!!

[mimugmail]

Sounds like Options of the VM. Real Hardware doesnt have this problem

[Bocephus]

Just like the previous poster if i plug my laptop into the modem i get speeds approaching 1 Gbps.  When i run through OPNSENSE and plug directly into the LAN port it appears that OPNSENSE throttles the speeds to no more than 360 Gbps.

I've edited the LAN MSS with 1400 as recommended....but no changes.

Nothing special with OPNSENSE - Just a fresh install with the WAN and LAN ports.  No NAT, proxies, etc.

In testing I've tired a few other firewall packages (I hate them) but wanted to see if i was dealing with a hardware issue.  I've installed PFSENSE and have the same issues....throughput is throttled to about 360 Gbps.

However, i've installed IPFIRE - using the exact same hardware configuration and SURPRISE!  I'm able to achieve almost 1 Gbps throughput.

I'm confident this is some type of configuration setting......I love OPNSENSE but need to get this figured out.

Any help would be appreciated!!

I've had similar results.  I"m using AT&T 1gb Fiber, and recently switched from Untangle to OPNsense.  With Untangle I was regularly getting ~700 down, and ~900 up.  With OPNsense I'm getting ~350 down, and ~200 up.

I've tried clamping MSS to 1400, which seemed to slow it down slightly.  Then as someone else suggested I did a test with multiple streams, setting both upload and download to 8 streams, which was even worse at ~130 down and ~140 up (combined).

I"m not really sure where to go from here.

[jstubs99]

I too am experiencing this issue in Australia with an NBN HFC connection. I have a commercial firewall which i get > 900mbps but down to under 300 with opnsense.I have tried the MSS setting which slows it down slightly.

Its a real shame as it makes the solution unusable for me, if anyone has an ideas I'm open otherwise I guess its ipfire for me.

Cheers

[mimugmail]

https://docs.opnsense.org/troubleshooting/hardening.html

Did you check meltdown stuff?