Messages

My first thought was maybe shared forwarding, but you have this with pfsense 2.5 too, correct?

I have never tested pfSense 2.5.  As you had previously pointed out, my test was pfSense 2.4 which was FreeBSD 11.3 based.  I mistakenly looked at the version history page and mentioned it was FreeBSD 12.1 but we determined I was incorrect in my statement.

Ok, iflib, so it's related to 12.X-only, but strange it doesn't happen to vanilla 12.1

https://forums.freebsd.org/threads/what-is-kernel-if_io_tqg-100-load-of-core.70642/

Yeah, I saw that forum post when I was Googling around, too.  I don't know what is different than vanilla FreeBSD 12.1 and the OPNsense 20.7.2 kernel that makes it higher CPU usage but it is consistent in my testing every single time.

Do you still test with this hardware?
Dell T20 (Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (4 cores))

No, every single test, with the exception of that single test I did on the Dell T20 to see if more MHz helped, has been on a Dell R430.  I have several R430 that are like-for-like and I even ran different software on each one and the results were consistent to weed out that a X520 NIC or something was bad.  The results followed the OS/kernel installed regardless of which R430 I ran it on so I am fairly confident in my hardware.

[FreeBSD 12.1 (pf enabled):]

OK here are the test results as you requested:

FreeBSD 12.1 (pf enabled):

[root@fbsd1 ~]# uname -rv
12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC

[root@fbsd1 ~]# top -aSH
last pid:  2954;  load averages:  0.44,  0.42,  0.41                                                                      up 0+01:38:55  20:13:46
132 threads:   10 running, 104 sleeping, 18 waiting
CPU:  0.0% user,  0.0% nice, 19.7% system,  5.2% interrupt, 75.1% idle
Mem: 10M Active, 6100K Inact, 271M Wired, 21M Buf, 39G Free
Swap: 3968M Total, 3968M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        155 ki31      0    96K RUN      5  94:58  95.25% [idle{idle: cpu5}]
   11 root        155 ki31      0    96K CPU1     1  93:26  83.69% [idle{idle: cpu1}]
   11 root        155 ki31      0    96K RUN      0  94:44  73.68% [idle{idle: cpu0}]
   11 root        155 ki31      0    96K CPU4     4  93:15  72.51% [idle{idle: cpu4}]
   11 root        155 ki31      0    96K CPU3     3  93:36  64.80% [idle{idle: cpu3}]
   11 root        155 ki31      0    96K RUN      2  92:55  62.29% [idle{idle: cpu2}]
    0 root        -76    -      0   480K CPU2     2   0:05  34.76% [kernel{if_io_tqg_2}]
    0 root        -76    -      0   480K CPU3     3   0:14  33.49% [kernel{if_io_tqg_3}]
   12 root        -52    -      0   304K CPU0     0  26:23  29.62% [intr{swi6: task queue}]
    0 root        -76    -      0   480K -        4   0:05  23.31% [kernel{if_io_tqg_4}]
    0 root        -76    -      0   480K -        0   0:05  12.31% [kernel{if_io_tqg_0}]
    0 root        -76    -      0   480K -        1   0:04  10.01% [kernel{if_io_tqg_1}]
   12 root        -88    -      0   304K WAIT     5   3:55   2.28% [intr{irq264: mfi0}]
    0 root        -76    -      0   480K -        5   0:06   1.88% [kernel{if_io_tqg_5}]
2954 root         20    0    13M  3676K CPU5     5   0:00   0.02% top -aSH
   12 root        -60    -      0   304K WAIT     0   0:01   0.01% [intr{swi4: clock (0)}]
    0 root        -76    -      0   480K -        4   0:02   0.01% [kernel{if_config_tqg_0}]

Single Thread:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  8.45 GBytes  7.26 Gbits/sec  802             sender
[  4]   0.00-10.00  sec  8.45 GBytes  7.26 Gbits/sec                  receiver

10 Threads:
[ ID] Interval           Transfer     Bandwidth       Retr
[SUM]   0.00-10.00  sec  9.85 GBytes  8.46 Gbits/sec  2991             sender
[SUM]   0.00-10.00  sec  9.83 GBytes  8.45 Gbits/sec                  receiver


FreeBSD 12.1 with OPNsense Kernel (pf enabled):

[root@fbsd1 ~]# uname -rv
12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC

[root@fbsd1 ~]# fetch https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/sets/kernel-20.7.2-amd64.txz
[root@fbsd1 ~]# mv /boot/kernel /boot/kernel.old
[root@fbsd1 ~]# tar -C / -xf kernel-20.7.2-amd64.txz
[root@fbsd1 ~]# kldxref /boot/kernel
[root@fbsd1 ~]# reboot

[root@fbsd1 ~]# uname -rv
12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #0  b3665671c4d(stable/20.7)-dirty: Thu Aug 27 05:58:53 CEST 2020     root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP

[root@fbsd1 ~]# top -aSH
last pid: 43891;  load averages:  0.99,  0.49,  0.20                                                                      up 0+00:04:28  20:29:24
131 threads:   13 running, 100 sleeping, 18 waiting
CPU:  0.0% user,  0.0% nice, 62.5% system,  3.5% interrupt, 33.9% idle
Mem: 14M Active, 1184K Inact, 270M Wired, 21M Buf, 39G Free
Swap: 3968M Total, 3968M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
    0 root        -76    -      0   480K CPU3     3   0:08  81.27% [kernel{if_io_tqg_3}]
    0 root        -76    -      0   480K CPU1     1   0:09  74.39% [kernel{if_io_tqg_1}]
    0 root        -76    -      0   480K CPU5     5   0:08  73.20% [kernel{if_io_tqg_5}]
    0 root        -76    -      0   480K CPU0     0   0:21  71.79% [kernel{if_io_tqg_0}]
   11 root        155 ki31      0    96K RUN      4   4:09  54.15% [idle{idle: cpu4}]
   11 root        155 ki31      0    96K RUN      2   4:09  51.30% [idle{idle: cpu2}]
    0 root        -76    -      0   480K CPU2     2   0:05  40.10% [kernel{if_io_tqg_2}]
    0 root        -76    -      0   480K -        4   0:09  37.60% [kernel{if_io_tqg_4}]
   11 root        155 ki31      0    96K RUN      0   4:03  26.48% [idle{idle: cpu0}]
   11 root        155 ki31      0    96K RUN      5   4:14  25.87% [idle{idle: cpu5}]
   11 root        155 ki31      0    96K RUN      1   4:09  24.32% [idle{idle: cpu1}]
   12 root        -52    -      0   304K RUN      2   1:12  20.63% [intr{swi6: task queue}]
   11 root        155 ki31      0    96K CPU3     3   4:00  17.30% [idle{idle: cpu3}]
   12 root        -88    -      0   304K WAIT     5   0:10   1.47% [intr{irq264: mfi0}]
43891 root         20    0    13M  3660K CPU4     4   0:00   0.03% top -aSH
   21 root        -16    -      0    16K -        4   0:00   0.02% [rand_harvestq]
   12 root        -60    -      0   304K WAIT     1   0:00   0.02% [intr{swi4: clock (0)}]

Single Thread:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  2.89 GBytes  2.48 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  2.89 GBytes  2.48 Gbits/sec                  receiver

10 Threads:
[ ID] Interval           Transfer     Bandwidth       Retr
[SUM]   0.00-10.00  sec  8.16 GBytes  7.01 Gbits/sec  4260             sender
[SUM]   0.00-10.00  sec  8.13 GBytes  6.98 Gbits/sec                  receiver


I included the "top -aSH" output again because my general observation between OPNsense kernel and FreeBSD 12.1 stock kernel is the "[kernel{if_io_tqg_X}]" process usage.  Even on an actual OPNsense 20.7.2 installation I notice the exact same behavior of the "[kernel{if_io_tqg_X}]" being consistently higher and throughput significantly slower, specifically on single threaded tests.  Note that both of the top outputs were only from the 10 thread count tests only as I did not think to capture them during the single threaded test.

I can't help but think that whatever high "[kernel{if_io_tqg_X}]" on the OPNsense kernel means is starving the system of throughput potential.

Thoughts?  Next steps I can run and provide results from?

Oh, I guess I misunderstood franco's instructions I thought they were asking me to drop the 20.7.2 kernel linked on top/in place on my FreeBSD 12.1 install which I was asking how exactly to do that.

I think with your clarification and re-reading the post, franco was just asking me to try an install of 20.7.2, which happens to be running that kernel, and re-run my tests to see if it improves.

If that's the case, I will try and report back my findings with OPNsense 20.7.2.

I am not super familiar with FreeBSD so how would I go about swapping your kernel in for the existing stock FreeBSD 12.1 one I am running?  I searched around on Google and I found how to build a customer kernel from source but this txz file you linked appears to be already compiled so I don't think that's what I want to do.

I also found reference to pkg-static to install locally downloaded packages but wanted to get some initial guidance before totally hosing this up.

[Test Scenario 1:]

OK, back to basics here.  I couldn't leave well enough alone and I did more testing tonight because I just couldn't believe that my CPU couldn't even do single threaded gigabit.  Here's my test scenario:

Test Scenario 1:

• Physical Linux Server (CentOS 7) on VLAN 2 (iperf3 client)
• Virtual Linux Server (CentOS 7) on VLAN 24 (iperf3 server)
• Dell PowerEdge R430 w/Intel X520-SR2 and HardenedBSD 12-STABLE (BUILD-LATEST 2020-08-31)

Single Threaded:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.00 GBytes   863 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.00 GBytes   860 Mbits/sec                  receiver

6 Parallel Threads:
[ ID] Interval           Transfer     Bandwidth       Retr
[SUM]   0.00-10.00  sec  2.23 GBytes  1.91 Gbits/sec  938             sender
[SUM]   0.00-10.00  sec  2.22 GBytes  1.90 Gbits/sec                  receiver

Notice a common theme here with the ~850 Mbps single threaded test.  It's pretty close to what I get with OPNsense.  Note this is THROUGH the firewall and not from the firewall.  Also note my system did have IPv6 addresses from my ISP on each of the interfaces, though, I was only testing IPv4 traffic.

Test Scenario 2:

• Physical Linux Server (CentOS 7) on VLAN 2 (iperf3 client)
• Virtual Linux Server (CentOS 7) on VLAN 24 (iperf3 server)
• Dell PowerEdge R430 w/Intel X520-SR2 and FreeBSD 12.1-RELEASE

Single Threaded:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  9.75 GBytes  8.38 Gbits/sec  573             sender
[  4]   0.00-10.00  sec  9.75 GBytes  8.38 Gbits/sec                  receiver

6 Parallel Threads:
[ ID] Interval           Transfer     Bandwidth       Retr
[SUM]   0.00-10.00  sec  10.5 GBytes  9.05 Gbits/sec  3607             sender
[SUM]   0.00-10.00  sec  10.5 GBytes  9.04 Gbits/sec                  receiver

I couldn't believe my eyes as I had to do a triple check that it was in fact pushing 8.38 Gbps THROUGH the FreeBSD 12.1 server and it wasn't taking some magical alternate path somehow.  It was, in fact, going through the FreeBSD router.  As you can see, parallel test is about 1 Gbps less than wire speed.  Excellent!  Also note my system did have IPv6 addresses from my ISP on each of the interfaces, though, I was only testing IPv4 traffic.

I thought I would enable pfctl on the FreeBSD 12.1 router to see how that affected performance.  Not sure how much adding rules impacts throughput but I did notice a measurable drop in the single thread test (6.23 Gbps) but the parallel thread test was negligible (8.94 Gbps).

As of right now, it seems so so so strange to me that HardenedBSD exhibits the same exact single threaded throughput and likewise low parallel thread throughput over FreeBSD.

I am willing to accept that I am not accounting for something here; however, near wire speed throughput on the same exact hardware on FreeBSD versus HardenedBSD, it seems to me something is very different with HardenedBSD.

What are your thoughts?

I built a new OPNsense server on my spare Dell PowerEdge R430 server that has the same CPU in it as my one I am currently using.

I can confirm that the problem appears to be my CPU and/or hardware since the same exact NIC was moved from the Dell PowerEdge T20 which previously tested out at 7.53 Gbps to this R430 server and the test results are much lower:

[root@client1 ~]# iperf3 -c 192.168.1.31
...
[  4]   0.00-10.00  sec  2.13 GBytes  1.83 Gbits/sec   72             sender
[  4]   0.00-10.00  sec  2.13 GBytes  1.83 Gbits/sec                  receiver

[root@client1 ~]# iperf3 -c 192.168.1.31 -P 10
...
[SUM]   0.00-10.00  sec  4.78 GBytes  4.10 Gbits/sec  1143             sender
[SUM]   0.00-10.00  sec  4.75 GBytes  4.08 Gbits/sec                  receiver

One observation is on like-for-like hardware, the new R430 is performing more than double the throughput on the single thread test and more than a gigabit more on parallel test than my currently used R430 I have been experiencing problems with.  No idea why this is.

I guess I have a decision to make about buying a new CPU or a new server.

Fresh install of OPNsense 20.7 on a Dell T20 (Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (4 cores)):

[root@client1 ~]# iperf3 -c 192.168.1.31
...
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  8.29 GBytes  7.12 Gbits/sec    2             sender
[  4]   0.00-10.00  sec  8.29 GBytes  7.12 Gbits/sec                  receiver

[root@client1 ~]# iperf3 -c 192.168.1.31 -P 10
...
[ ID] Interval           Transfer     Bandwidth       Retr
[SUM]   0.00-10.01  sec  8.77 GBytes  7.53 Gbits/sec  139             sender
[SUM]   0.00-10.01  sec  8.77 GBytes  7.53 Gbits/sec                  receiver

It's just hard to believe that E3-1225 v3 @ 2.4GHz/3.2GHz versus an E5-2620 v3 3.2GHz/3.6GHz is that much difference for a single thread test; however, it's clear, the results don't lie.  There's either something wrong with my hardware, my install or it's just too slow of a CPU to push single threaded performance past about 850 Mbps.

And you're right about the pfSense version of FreeBSD.  I just double checked the page (https://docs.netgate.com/pfsense/en/latest/releases/versions-of-pfsense-and-freebsd.html) and, in spite of it being clearly marked 2.5.0 TBD, I didn't even pay attention that it definitely was not the edition I installed.

I mean of course running a parallel test is going to yield better results if the firewall has multi-core CPU(s) and you are maxing out a CPU core.

The issue I have is that that single threaded throughput is only about 850 Mbps on my non-virtualized hardware.  That seems not right to me but I only know my environment so I might just be wrong.

And yes, I did test through the firewall before I started doing tests from the firewall.  Through the firewall nets me similar performance for single threaded:

[root@client1 ~]# iperf3 -f m -c 192.168.1.31
...
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   973 MBytes   816 Mbits/sec   22             sender
[  4]   0.00-10.00  sec   970 MBytes   814 Mbits/sec                  receiver

And, as expected, increased throughput when running in parallel:

[root@client1 ~]# iperf3 -f m -c 192.168.1.31 -P 10
...
[ ID] Interval           Transfer     Bandwidth       Retr
...
[SUM]   0.00-10.00  sec  3.26 GBytes  2798 Mbits/sec  4464             sender
[SUM]   0.00-10.00  sec  3.23 GBytes  2776 Mbits/sec                  receiver

Can you humor me and run a single threaded test through your hardware and show me the output?

If OPNsense is truly not broken in this release then I guess my CPU core speed isn't enough to achieve what I am looking to do and I need to look on eBay for a faster one.  That being said, it appears there are several others reporting degraded performance since upgrading so maybe there is something to my claim.

Edit:  I see your single threaded non-IPS throughput is 6826 Mbps.  See, even your single threaded test absolutely crushes mine.  I get that your CPU is @ 3.7 GHz and a v6 but really, almost 7 Gbps versus less than my 1 Gbps.  I have a v3 Xeon that has a higher clock rate (maybe 3.2 GHz?) I can try to test out tomorrow to see what results I get.

[OPNsense 20.7.1 (amd64)]

OK so at the risk of seeming like I am only talking to myself at this point, I think I found a commonality amongst the poor performance -- it's OPNsense.

I built a fresh new and updated OPNsense 20.7.1 VM on VMware ESXi 6.7U3, imported my configuration backup from my physical server and re-mapped all the interfaces to the new vmx0_vlanX names and things are working, albeit even slower than the physical hardware:

root@opnsense1:~ # iperf3 -c 192.168.1.31
...
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   705 MBytes   591 Mbits/sec    0             sender
[  5]   0.00-10.41  sec   705 MBytes   568 Mbits/sec                  receiver

Seems pretty awful.  So I decided to create a two new OPNsense 20.7.1 VMs and configure one as a VLAN trunk and the other as non-trunk to test if the problem lied within the VLAN implementation itself:

OPNsense 20.7.1 (amd64)

VLAN and pf Enabled:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   949 MBytes   796 Mbits/sec    0             sender
[  5]   0.00-10.40  sec   949 MBytes   766 Mbits/sec                  receiver

VLAN and pf Disabled (pfctl -d):
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.22 GBytes  1.05 Gbits/sec    0             sender
[  5]   0.00-10.41  sec  1.22 GBytes  1.01 Gbits/sec                  receiver

Non-VLAN and pf Enabled:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   854 MBytes   716 Mbits/sec    0             sender
[  5]   0.00-10.40  sec   854 MBytes   688 Mbits/sec                  receiver

Non-VLAN and pf Disabled (pfctl -d):
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   983 MBytes   825 Mbits/sec    0             sender
[  5]   0.00-10.40  sec   983 MBytes   793 Mbits/sec                  receiver

As you can see, the VLAN trunk configured VM had slightly better performance.  Perhaps environmental impacts caused the performance differences as I would expect them to be nearly the same.  Even at the differences I'm seeing, I would consider it mostly negligible given the link is 10 gigabit.  I also tested without pf to see if the throughput was measurable.  Both tests show that it is in fact better without pf, though, kinda pointless to have a network perimeter firewall without it running...

Next I thought maybe this is just a fluke and all three OPNsense servers just suck on VMware ESXi and dislike the hardware or configuration or maybe my ESX host just can't push traffic.  I had a CentOS 8.2.2004 VM already deployed and configured on the same network segment I had been testing on so I loaded up iperf3 on it to see if it was an ESX host/network problem.

CentOS 8.2.2004 (x86_64)

Non-VLAN and firewalld Enabled:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.7 GBytes  9.17 Gbits/sec   11             sender
[  5]   0.00-10.04  sec  10.7 GBytes  9.14 Gbits/sec                  receiver

Non-VLAN and firewalld Disabled:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.8 GBytes  9.32 Gbits/sec    1             sender
[  5]   0.00-10.04  sec  10.8 GBytes  9.28 Gbits/sec                  receiver

Tested with firewall on and off just for fun to see how much iptables slowed the Linux test down.  As you can see, 9.14 Gbps to 9.32 Gbps on this test.  The problem isn't my ESX host or my network.

I then thought it might be a BSD problem.  Perhaps something with running inside VMware or the vmxnet3 driver that is problematic.  I tried to figure out how to install HardenedBSD but it seemed too difficult difficult as my quick search for an ISO yielded not much.  As such, I used FreeBSD.  Hopefully it's close enough!

FreeBSD 12.1 (amd64)

VLAN and pf Disabled (not configured):
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.35 Gbits/sec    0             sender
[  5]   0.00-10.42  sec  10.9 GBytes  8.97 Gbits/sec                  receiver

Non-VLAN and pf Disabled (not configured):
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.36 Gbits/sec   13             sender
[  5]   0.00-10.21  sec  10.9 GBytes  9.17 Gbits/sec                  receiver

I thought I hadn't spent enough time already dorking around with this so why not configure one test VM to be VLAN trunking and the other not to see if there are any differences.  As you can see, FreeBSD 12.1 pushed the packets, fast, regardless of VLAN or otherwise.  Problem doesn't seem to be vmxnet3/ESXi and FreeBSD related.

Finally, I came to the conclusion that maybe OPNsense 20.7 is just broken.  As such, I loaded up a OPNsense 19.7 test VM and gave it a go.

OPNsense 19.7.10_1 (amd64)

Non-VLAN and pf Enabled:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.75 GBytes  1.50 Gbits/sec    0             sender
[  5]   0.00-10.44  sec  1.75 GBytes  1.44 Gbits/sec                  receiver

Non-VLAN and pf Disabled (pfctl -d):
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.57 GBytes  2.21 Gbits/sec    0             sender
[  5]   0.00-10.48  sec  2.57 GBytes  2.11 Gbits/sec                  receiver


Not good.  You can see the results of 1.75 Gbps to 2.57 Gbps is measurably better than my test results with OPNsense 20.7 but nowhere near stellar.  I was very much over testing at this point so I opted not to do a VLAN versus non-VLAN configuration.  That being said, based on historical results, I am sure that the difference in results would have been negligible.

To add to this, as a general observation, whenever the iperf3 test is running on OPNsense, a constant ping of the firewall starts to drop packets like it is choked out and cannot keep up.  I did not experience this at all on CentOS or FreeBSD when testing.

Why is OPNsense so bad at throughput in my tests?  If it's not, what am I doing wrong?  The commonality amongst these tests seems to be OPNsense, regardless if it's 19.7 or 20.7, though, the former is better than the later.

Edit:  Because why not at this point.  Let's test pfSense!

pfSense 2.4.5 (amd64)

Non-VLAN and pf Enabled:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.80 GBytes  3.26 Gbits/sec   67             sender
[  5]   0.00-10.26  sec  3.80 GBytes  3.18 Gbits/sec                  receiver

Non-VLAN and pf Disabled (pfctl -d):
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.66 GBytes  4.86 Gbits/sec  109             sender
[  5]   0.00-10.22  sec  5.66 GBytes  4.76 Gbits/sec                  receiver

pfSense is not stellar, especially considering it is based on FreeBSD 12.1 and I tested FreeBSD 12.1 and got very different (better) results.  That being said, both results are much, much faster than any OPNsense test I could push regardless if physical or virtual.

Edit 2:  Fixed a typo in my comments where I erroneously used 20.1 instead of 20.7 when referring to editions of OPNsense.

TL;DR:  OPNsense seems to be dog slow compared to FreeBSD 12.1 and CentOS 8.2 at raw network throughput.  What gives?  What am I doing wrong that it can be this huge of a performance gap?

Just a status update:

Swapped optics on the switch side (both have now been switched) and swapped for a new fiber patch cable.  Same results.  I also re-enabled "Hardware CRC" and "VLAN Hardware Filtering" but left "Hardware TSO" and "Hardware LRO" disabled as I read most drivers are broken for those functions.

I also added this to /boot/loader.conf.local and rebooted:

hw.cxgbe.toecaps_allowed=0
hw.cxgbe.rdmacaps_allowed=0
hw.cxgbe.iscsicaps_allowed=0
hw.cxgbe.fcoecaps_allowed=0

Absolutely zero impact in performance.  Tomorrow I think I'll unbox my other PowerEdge R430 and put the original Intel X520-SR2 NIC in it and see if I can duplicate the problem.

I am at a total loss of what is going on here.

I know that the Broadcom drivers aren't the best but I figured it was worth a test.  That being said, I just swapped the Intel X520-SR2 with a Chelsio T540-CR which seems to have excellent FreeBSD support and that family of NICs seems frequently recommended.

Here's the results from the Chelsio T540-CR:

# iperf3 -c 192.168.1.31
Connecting to host 192.168.1.31, port 5201
[  5] local 192.168.1.1 port 19465 connected to 192.168.1.31 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   112 MBytes   943 Mbits/sec    0   8.00 MBytes
[  5]   1.00-2.00   sec   110 MBytes   924 Mbits/sec    0   8.00 MBytes
[  5]   2.00-3.00   sec   112 MBytes   939 Mbits/sec    0   8.00 MBytes
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec    0   8.00 MBytes
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec    0   8.00 MBytes
[  5]   5.00-6.00   sec   112 MBytes   939 Mbits/sec    0   8.00 MBytes
[  5]   6.00-7.00   sec   112 MBytes   940 Mbits/sec    0   8.00 MBytes
[  5]   7.00-8.00   sec   112 MBytes   938 Mbits/sec    0   8.00 MBytes
[  5]   8.00-9.00   sec   112 MBytes   940 Mbits/sec    0   8.00 MBytes
[  5]   9.00-10.00  sec   112 MBytes   940 Mbits/sec    0   8.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec    0             sender
[  5]   0.00-10.32  sec  1.09 GBytes   909 Mbits/sec                  receiver

Also thought it was interesting there were zero retransmits on the test.

I swapped out the optic on the NIC when I swapped the NIC itself.  I will swap the optic on the switch and maybe try a different switch port and fiber patch cable tomorrow, though, I doubt those are the issue.

Unfortunately, it appears that the issue was not my Intel X520-SR2 NIC as the Chelsio T540-CR exhibits the same behavior.

To add to this, I re-configured all my VLANs on bge0 (onboard NIC) and moved all my interfaces over to each respective bge0_vlanX interface and re-ran my iperf3 tests.

On my first test, I got the same throughput as with my Intel X520-SR2 NIC:

# iperf3 -c 192.168.1.31
Connecting to host 192.168.1.31, port 5201
[  5] local 192.168.1.1 port 42455 connected to 192.168.1.31 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  92.0 MBytes   772 Mbits/sec   91   5.70 KBytes
[  5]   1.00-2.00   sec  91.1 MBytes   764 Mbits/sec   88    145 KBytes
[  5]   2.00-3.00   sec  86.1 MBytes   722 Mbits/sec   86    836 KBytes
[  5]   3.00-4.00   sec  92.5 MBytes   776 Mbits/sec   76    589 KBytes
[  5]   4.00-5.00   sec   107 MBytes   894 Mbits/sec    0    803 KBytes
[  5]   5.00-6.00   sec   107 MBytes   898 Mbits/sec    2    731 KBytes
[  5]   6.00-7.00   sec   109 MBytes   914 Mbits/sec    1    658 KBytes
[  5]   7.00-8.00   sec   110 MBytes   926 Mbits/sec    0    863 KBytes
[  5]   8.00-9.00   sec   107 MBytes   898 Mbits/sec    2    748 KBytes
[  5]   9.00-10.00  sec   109 MBytes   918 Mbits/sec    1    663 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1011 MBytes   848 Mbits/sec  347             sender
[  5]   0.00-10.32  sec  1010 MBytes   821 Mbits/sec                  receiver

For reference, I just tested with my MacBook Pro against the same iperf3 server and was able to push 926 Mbps and re-tested my QNAP to QNAP transfer and it did 9.39 Gbps to completely rule out it's an iperf3 server thing.

For the sake of testing because why not, I re-ran iperf3 from my OPNsense server once more and got near gigabit throughput:

# iperf3 -c 192.168.1.31
Connecting to host 192.168.1.31, port 5201
[  5] local 192.168.1.1 port 8283 connected to 192.168.1.31 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   108 MBytes   906 Mbits/sec    0    792 KBytes
[  5]   1.00-2.00   sec   111 MBytes   932 Mbits/sec    2    698 KBytes
[  5]   2.00-3.00   sec   111 MBytes   930 Mbits/sec    1    638 KBytes
[  5]   3.00-4.00   sec   108 MBytes   905 Mbits/sec    1    585 KBytes
[  5]   4.00-5.00   sec   111 MBytes   929 Mbits/sec    0    816 KBytes
[  5]   5.00-6.00   sec   111 MBytes   929 Mbits/sec    1    776 KBytes
[  5]   6.00-7.00   sec   111 MBytes   928 Mbits/sec    1    725 KBytes
[  5]   7.00-8.00   sec   108 MBytes   906 Mbits/sec    2    663 KBytes
[  5]   8.00-9.00   sec   111 MBytes   928 Mbits/sec    2    616 KBytes
[  5]   9.00-10.00  sec   111 MBytes   928 Mbits/sec    0    837 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.07 GBytes   922 Mbits/sec   10             sender
[  5]   0.00-10.32  sec  1.07 GBytes   892 Mbits/sec                  receiver

One thing I noticed between the first and second iperf3 test was the "Retr" column of 347 vs 10.  I researched what that meant for iperf3 and found this: "It's the number of TCP segments retransmitted. This can happen if TCP segments are lost in the network due to congestion or corruption."

I also noticed during my second iperf3 test that there was now a kernel process using 99.81% CPU:

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        155 ki31      0   192K CPU3     3   9:02 100.00% [idle{idle: cpu3}]
    0 root        -92    -      0   848K CPU2     2   0:30  99.81% [kernel{bge0 taskq}]

Additionally, I am not sure "Retr" in itself is a smoking gun as the QNAP to QNAP test that yielded 9.39 Gbps did 2218 retries.

The search continues.

I originally posted on Reddit but figured I might get more traction here with this.

I have an OPNsense 20.7.1 server running on a Dell R430 with 16 GB DDR4 RAM, an Intel Xeon E5-2620 v3 (6 cores/12 threads @ 2.40GHz) CPU and an Intel X520-SR2 10GbE NIC.

My network has several VLANs and network subnets with my OPNsense router functioning as a router on a stick doing all the traffic firewalling and routing between each network segment.

I recently upgraded my OPNsense to 20.7.1 and on a whim decided to run an iperf3 test between two VMs on different network segments to see what kind of throughput I was getting. I am certain, at least at some point, this very same hardware pushed over 6 Gbps on the same iperf3 test. Today it was getting around 850 Mbps every single time.

I started iperf3 as a server on my QNAP NAS device which is also attached to the same 10 Gbps switch and ran iperf3 as a client from OPNsense on the same network segment and got the same 850 Mbps throughput.

To make sure I wasn't limited by the QNAP NAS device, I ran the same iperf3 test with my other QNAP NAS device as a client to the first QNAP NAS device and it pushed 8.6 Gbps across the same network segment (no OPNsense involved) so both the QNAP and the switch can push it.

My question is what do I have going wrong here? Even the same network segment, OPNsense can't do more than 850 Mbps throughput. I have no idea if this was happening pre-upgrade to 20.7.1 but I know for sure it is happening now. I would assume an iperf3 test from the OPNsense server on the same network segment would surely remove any doubt it was firewalling, etc.

The interface shows 10 Gbps link speed, too, both from ifconfig and the switch itself.

My current MBUF Usage is 1 % (17726/1010734).

IDS/IPS package is installed but disabled.

I had "Hardware CRC" and "Hardware TSO" and "Hardware LRO" and "VLAN Hardware Filtering" all enabled. I have since set those all to disabled and rebooted. I can confirm that it disabled by looking at the interface flags in ifconfig:

Pre-reboot:
options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>

Post-reboot:
options=803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC>

I ran top and was able to see a process (kernel{if_io_tqg_2}) utilize near 100% of a CPU core during this iperf3 test:

# top -aSH

last pid: 22772;  load averages:  1.23,  0.94,  0.79                                                                                                                                                                      up 5+23:48:52  14:24:22
233 threads:   15 running, 193 sleeping, 25 waiting
CPU:  1.0% user,  0.0% nice, 16.1% system,  0.5% interrupt, 82.4% idle
Mem: 1485M Active, 297M Inact, 1657M Wired, 935M Buf, 12G Free
Swap: 8192M Total, 8192M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
    0 root        -76    -      0   848K CPU2     2 279:51  99.77% [kernel{if_io_tqg_2}]
   11 root        155 ki31      0   192K CPU3     3 130.8H  98.78% [idle{idle: cpu3}]
   11 root        155 ki31      0   192K CPU9     9 131.3H  98.75% [idle{idle: cpu9}]
   11 root        155 ki31      0   192K CPU1     1 129.7H  98.68% [idle{idle: cpu1}]
   11 root        155 ki31      0   192K CPU10   10 138.1H  98.33% [idle{idle: cpu10}]
   11 root        155 ki31      0   192K CPU5     5 130.5H  97.51% [idle{idle: cpu5}]
   11 root        155 ki31      0   192K CPU0     0 138.3H  95.78% [idle{idle: cpu0}]
   11 root        155 ki31      0   192K CPU8     8 137.7H  95.25% [idle{idle: cpu8}]
   11 root        155 ki31      0   192K CPU6     6 138.7H  95.20% [idle{idle: cpu6}]
   11 root        155 ki31      0   192K CPU4     4 138.4H  94.26% [idle{idle: cpu4}]
22772 root         82    0    15M  6772K CPU7     7   0:04  93.83% iperf3 -c 192.168.1.31
   11 root        155 ki31      0   192K RUN      7 129.4H  68.75% [idle{idle: cpu7}]
   11 root        155 ki31      0   192K RUN     11 126.8H  46.12% [idle{idle: cpu11}]
    0 root        -76    -      0   848K -        4 277:00   5.12% [kernel{if_io_tqg_4}]
   12 root        -60    -      0   400K WAIT    11 449:21   5.02% [intr{swi4: clock (0)}]
    0 root        -76    -      0   848K -        8 317:40   3.81% [kernel{if_io_tqg_8}]
    0 root        -76    -      0   848K -        0 272:13   2.71% [kernel{if_io_tqg_0}]

I occasionally see flowd_aggregate.py pop up to 100% but it doesn't seem consistent or relevant to when iperf3 is running:

# top -aSH

last pid: 99781;  load averages:  1.15,  0.90,  0.77                                                                                                                                                                      up 5+23:47:27  14:22:57
232 threads:   14 running, 193 sleeping, 25 waiting
CPU:  8.5% user,  0.0% nice,  1.6% system,  0.4% interrupt, 89.5% idle
Mem: 1481M Active, 299M Inact, 1656M Wired, 935M Buf, 12G Free
Swap: 8192M Total, 8192M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
43465 root         90    0    33M    25M CPU7     7   7:11  99.82% /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.7)
   11 root        155 ki31      0   192K CPU9     9 131.3H  99.80% [idle{idle: cpu9}]
   11 root        155 ki31      0   192K CPU3     3 130.8H  99.68% [idle{idle: cpu3}]
   11 root        155 ki31      0   192K CPU10   10 138.1H  99.50% [idle{idle: cpu10}]
   11 root        155 ki31      0   192K CPU6     6 138.7H  98.53% [idle{idle: cpu6}]
   11 root        155 ki31      0   192K RUN      5 130.5H  98.20% [idle{idle: cpu5}]
   11 root        155 ki31      0   192K CPU1     1 129.7H  97.97% [idle{idle: cpu1}]
   11 root        155 ki31      0   192K CPU11   11 126.8H  96.52% [idle{idle: cpu11}]
   11 root        155 ki31      0   192K CPU0     0 138.3H  96.43% [idle{idle: cpu0}]
   11 root        155 ki31      0   192K CPU8     8 137.7H  95.95% [idle{idle: cpu8}]
   11 root        155 ki31      0   192K CPU2     2 138.3H  95.81% [idle{idle: cpu2}]
   11 root        155 ki31      0   192K CPU4     4 138.4H  93.94% [idle{idle: cpu4}]
   12 root        -60    -      0   400K WAIT     4 449:17   5.10% [intr{swi4: clock (0)}]
    0 root        -76    -      0   848K -        4 276:55   4.95% [kernel{if_io_tqg_4}]

What is going on here?