Very slow WAN bandwidth

[peezy]

Greetings all,

For the past couple of days, I have been troubleshooting my new Opnsense install. I am experiencing around 10-20 percent of my max downstream bandwidth (1 gbit).

I am running an Seedstudio Odyssey with a J4125 celeron, and 2x intel i211 NICs. I have read that there were previous issues with the driver for this NIC, and that said issues should be resolved. I agree with this due to my iPerf test results on the LAN side, which I will get to in a bit.

My ISP connection is through Comcast. When connecting directly to the Docsis 3.1 modem, I get around 700Mbps.

I have tested via Iperf from the LAN to the LAN port, and from the LAN out to a public Iperf server .
LAN to LAN:
Client connecting to 10.0.0.254, TCP port 5201
TCP window size:  153 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.15 port 43978 connected with 10.0.0.254 port 5201
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   861 MBytes   722 Mbits/sec
par_admin@pflixv4:~$ iperf -c 10.0.0.254 -p 5201
------------------------------------------------------------
Client connecting to 10.0.0.254, TCP port 5201
TCP window size:  357 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.15 port 44360 connected with 10.0.0.254 port 5201
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   542 MBytes   454 Mbits/sec
par_admin@pflixv4:~$ iperf -c 10.0.0.254 -p 5201
------------------------------------------------------------
Client connecting to 10.0.0.254, TCP port 5201
TCP window size:  306 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.15 port 44490 connected with 10.0.0.254 port 5201
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   786 MBytes   660 Mbits/sec

LAN to WAN
par_admin@pflixv4:~$ iperf -c speedtest.serverius.net -p 5002
------------------------------------------------------------
Client connecting to speedtest.serverius.net, TCP port 5002
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.15 port 39488 connected with 178.21.16.76 port 5002
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.2 sec  50.0 MBytes  41.3 Mbits/sec

$ iperf3 -c iperf.scottlinux.com -p 5201
Connecting to host iperf.scottlinux.com, port 5201
[  5] local 2001:558:6022:29:addb:9c59:ef66:e2aa port 48795 connected to 2600:3c01::f03c:91ff:fed5:ed33 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.52 MBytes  46.3 Mbits/sec    0   2.45 MBytes
[  5]   1.00-2.00   sec  4.97 MBytes  41.7 Mbits/sec    0   3.00 MBytes
[  5]   2.00-3.00   sec  5.00 MBytes  42.0 Mbits/sec    0   3.00 MBytes
[  5]   3.00-4.00   sec  4.97 MBytes  41.6 Mbits/sec    0   3.00 MBytes
[  5]   4.00-5.00   sec  5.00 MBytes  42.0 Mbits/sec    0   3.00 MBytes
[  5]   5.00-6.00   sec  4.94 MBytes  41.4 Mbits/sec    0   3.00 MBytes
[  5]   6.00-7.00   sec  4.98 MBytes  41.8 Mbits/sec    0   3.00 MBytes
[  5]   7.00-8.00   sec  5.02 MBytes  42.1 Mbits/sec    0   3.00 MBytes
[  5]   8.00-9.00   sec  4.94 MBytes  41.4 Mbits/sec    0   3.00 MBytes

Speedtest.net ~100Mbits/s
Google Speedtest ~ 220Mbits/s

I dont use IDS/IPS, but I do use Sensei. Though I found that when disabling Sensei that the speed tests did not change much. I also use ClamAV, and have the Netflow service (samplicate) and insight aggregator running.

CPU wise, I see spikes here and there, but the usage hovers around 10-20 percent when conducting WAN tests.
I do get one CPU maxed out to 100 percent when doing the LAN test. I have not been able to get closer to a gigabit on the LAN side, so its possible there is some bottleneck there as well.
I have looked over the tunables, but other than setting hw.ibrs_disable to 1, the others I havent touched.

[peezy]

Bump?
Should I post this in a different forum?

[testo_cz]

@peezy

IMHO, you are in the right forum. Its just that there are many posts like yours every month...

Please search over the forum more and you'll find some hints, I'm sure.

I recall i211/i210 were discussed here sometime ago. Tunables you have under dev.igb.0 for the first igb driver NIC. Maybe dev.em.0 , I don't run i211 NIC personally.

There is FreeBSD network tuning guide.

What is worth to check is cpu frequency , to determine power management state might be throttling (sysctl -a | grep cpu | grep freq )

Public iperf servers were discussed here too. Seems they differ in performance. Generally , one must do the tests with multiple TCP streams (, e.g. iperf3 -P5 )

Some people have to deal with MTU on WAN .

Good luck
T.