Bandwidth slow through firewall

[Ric878]

Hi,

I recently upgraded to Gig internet with Comcast and have been having trouble getting my full speed when traffic is going through the firewall.

To do some baseline testing, I have connected my laptop directly to the cable modem and when I do a speed test on http://speedtest.xfinity.com/ I consistently get between 900 - 940 Mbps. When I do the same speed test with OPNsense as the firelwall/router, my download speeds are ~ 600 Mbps. I have an IPSec vpn running but I have removed it and run the speed tests and still see the same results. For testing pruposes, I have all other devices removed from the switch so it's just my laptop, the OPNsense box, and the cable modem. The laptop is connected via ethernet cable during the tests. If I remove the OPNsense box from the equation, I consistently get the faster 900 - 940 Mbps speed test results.

I have done an iperf3 test from my laptop to the OPNsense LAN port and am getting ~940 Mbps test results, that of course is not traversing the firewall.

Here are the OPNsense box specs:
Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz (4 cores)
8 Gigs of Ram
I350 Quad Port Network Card

Modem:
Motorola MB8600

Any troubleshooting tips would be appreciated.

Thanks,
Ric

[mimugmail]

Set MSS on LAN to 1400

[Ric878]

Set MSS on LAN to 1400

Tried that, no change. Thanks for the suggestion though.

[mimugmail]

IPS and Proxy disabled?

[Ric878]

IPS and Proxy disabled?

Yes, both are disabled.

[youngman]

If I remove the OPNsense box from the equation, I consistently get the faster 900 - 940 Mbps speed test results.

When you add the OPNsense box, is the modem/connection interface bridged properly? No double NATing or other double-ups?

[Ric878]

If I remove the OPNsense box from the equation, I consistently get the faster 900 - 940 Mbps speed test results.

When you add the OPNsense box, is the modem/connection interface bridged properly? No double NATing or other double-ups?

No double NATing. As a matter of fact, the modem doesn't have any router or firewall features, other than passing a local IP during boot before it passes a public IP. For reference, it is a Motoroal MB8600.

Thanks,
Ric

[Ric878]

Just wanted to update with some additional steps I've taken without success. So far I've tried to enable "hardware checksum offload", and "hardware TCP segmentation offload", rebooted and still had no improvement. I left "hardware large receive offload" disabled during testing.

I have also replaced all Ethernet cables just to be sure that wasn't part of the issue.

[youngman]

Perhaps check out https://www.freebsd.org/cgi/man.cgi?tuning

dcol also has a more specific tuning thread here. I found it helpful: https://forum.opnsense.org/index.php?topic=6590.0

[Ric878]

Perhaps check out https://www.freebsd.org/cgi/man.cgi?tuning

dcol also has a more specific tuning thread here. I found it helpful: https://forum.opnsense.org/index.php?topic=6590.0

@youngman Thanks for those links. I'm playing around with some of the tunables now. I'll report back if there's any improvement.

[compunction]

@Ric878, you and I are in the same boat brother.  I have not found a solution yet, but been tinkering for a few weeks.

To rule out routing, IPS, etc...I have been testing with a python script so only the WAN interface is in scope.  If I plug my MacBook into the MB8600 I get between 930-950.  With my OPNsense box I see a max of 600-630.

This test shows only 428.87, but its the middle of the day ;-)
compunction@opnsense:~ % ./speedtest-cli.py
Retrieving speedtest.net configuration...
Testing from Comcast Cable (73.237.238.141)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Comcast (Atlanta, GA) [20.61 km]: 12.635 ms
Testing download speed................................................................................
Download: 428.87 Mbit/s
Testing upload speed................................................................................................
Upload: 41.34 Mbit/s
compunction@opnsense:~ %

A few things I have tried:
net.inet.tcp.tso="0"
dev.igb.0.fc="0"
dev.em.0.fc="0"
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
hw.em.rx_process_limit="-1"
hw.em.tx_process_limit="-1"
dev.igb.0.eee_disabled="1"
machdep.hyperthreading_allowed="0"
hw.igb.txd="2048"
hw.igb.rxd="2048"
hw.em.txd="2048"
hw.em.rxd="2048"
net.link.ifqmaxlen="4096"
hw.igb.max_interrupt_rate="32000"
hw.em.max_interrupt_rate="32000"
net.inet.tcp.sendbuf_inc="32768"
net.inet.tcp.recvbuf_inc="32768"
net.inet.tcp.sendbuf_max="4194304"
net.inet.tcp.recvbuf_max="4194304"

I have also tried swapping cat5e cables...no luck.

Still in the search for a resolution :).

[Ric878]

@compunction, I'm still playing with different combinations of settings with mixed results. Sometimes, I feel like I find a combination that seems to improve things, but the results don't seem to last. So, I am starting to think that my test methodology is flawed.

First, using speed test sites seems to be very inconsistent. I'm thinking that the only real way to get an accurate speed test is to do an iperf test with two computers, on either side of the OPNSense router (LAN/WAN). To be honest, I feel I may even be getting throttled from hitting the speed test sites so often. Just a paranoid theory really.

I'll report back once I get more info and setup a proper test. That said, what are the specs of your OPNSense box?

[compunction]

Intel(R) Core(TM) i7-8809G CPU @ 3.10GHz (3096.15-MHz K8-class CPU)
real memory  = 34359738368 (32768 MB)
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k>
em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k>

I have wondered myself if they could be throttling me, but when I test speed test on two different sites from two different machines, I see the speed drop as I would expect.  But general internet latency is likely a factor as well.  I need to grab my old hardware and setup a iperf box on the wan side.  I could likely put a 192.168.100.x IP on it has I have a route for that to get the router interface. (since I only have one WAN IP).

[Ric878]

Okay, so I did some new tests with iperf3 but I need someone to validate the setup and let me know if this is a valid test.

My OPNSense box has a 4 port network card in it. What I did was create a VLAN on my switch (VLAN 10), and created a new VLAN interface in OPNSense and attached it to one of my unused ports. I then connected this port to a switch port that was tagged for VLAN 10.

I also setup a spare PC with Debian 8 (PC2) on it and connected it to a static port on the switch for VLAN 10. I verified that without the OPNSense box in the middle, that I could not access the PC2 on VLAN10 from my test PC (PC1) on the default VLAN.

I then added the proper firewall rules on OPNSense to allow access to PC2 from PC1. All is good at this point. I proceeded to run my iperf3 test with PC1 as the client and PC2 as the server. My results were fantastic, ~915 Mbits/s

This is where I need someone to validate my test, my thinking is that because PC2 is on a separate VLAN that my OPNSense box had to route and allow all the iperf traffic through the firewall, proving that the OPNSense box was very capable of near 1 Gbs traffic. Question is, was my test valid. Only thing I can think about is that there was no NAT involved in the test. Would that make a significant enough difference in this validation?

[mimugmail]

Why not enabling NAT for testing?

Firewall : NAT : Outbound

Set to manual or hybrid, then add a rule, source PC1 LAN, dest PC2 LAN, translated address PC2 address.