OPNsense installation on ZFS using stock FreeBSD kernel - why so slow?

[AveryFreeman]

Hey,

I was reading this thread: https://forum.opnsense.org/index.php?topic=18754.15 which prompted me to do some throughput tests

I hadn't even noticed that my 10GbE iperf3 speeds were 6-8Gbps slower on my OPNsense VM than the rest of my VMs, until I did some testing - my iperf3 speeds were around 1Gbps for OPNsense 20.7.3 whereas they're consistently stuck at 9.4Gbps between everything else when there's no competing network traffic.

So I thought I'd try a different kernel.  I'd been wanting to do an installation over zfs as kind of a proof of concept, and also because I miss the zfs from my old pfsense installations (that's about all I miss), and since I was wanting to experiment with the kernel, I thought it'd be good to have snapshot capabilities so I could roll back easily if I wanted to.

I installed 20.7.3 over FreeBSD 12.1-RELEASE on a 24GB slc zmirror on this old j1800 shitbox I built from a Lenovo H500s motherboard (literally $15) using opnsense-bootstrap.  I replaced the kernel w/ the previously installed freebsd kernel after opnsense-bootstrap finished, and rebooted with it. 

This box used to get a pretty consistent 650Mbps with previous builds of OPNsense on my 1Gb comcast residential in speedtest-cli. Now it's rarely over 500Mbps, usually closer to 450.  And I'm just wondering:

What do people w/ experience with either think it is?  Does zfs tend to make systems with low processing power more sluggish? (It's got 8GB of ram, btw)  Or are there kernel optimizations for networking that the stock freebsd kernel is missing?

My arc_max is stock, which puts it up near the full 8GB, does anyone think if I tuned that, or my meta cache, I could get the speeds back up to 650Mbps again?

What about old OPNsense kernel?  I know the people in that thread said they got better speeds in ver. <19

Thanks

[AveryFreeman]

I have some iperf3 scores from a debian bullseye box on the same network:

Freebsd 12.1 Kernel on J1800 12.1 zfs installation (stock)

Code: [Select]

avery@debbox:~$ iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.39 port 39364 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  68.6 MBytes   575 Mbits/sec    0    281 KBytes       
[  5]   1.00-2.00   sec  65.6 MBytes   550 Mbits/sec    0    281 KBytes       
[  5]   2.00-3.00   sec  65.4 MBytes   549 Mbits/sec    0    281 KBytes       
[  5]   3.00-4.00   sec  65.2 MBytes   547 Mbits/sec    0    281 KBytes       
[  5]   4.00-5.00   sec  64.6 MBytes   542 Mbits/sec    0    281 KBytes       
[  5]   5.00-6.00   sec  64.9 MBytes   545 Mbits/sec    0    281 KBytes       
[  5]   6.00-7.00   sec  63.9 MBytes   536 Mbits/sec    0    281 KBytes       
[  5]   7.00-8.00   sec  65.0 MBytes   545 Mbits/sec    0    281 KBytes       
[  5]   8.00-9.00   sec  64.9 MBytes   545 Mbits/sec    0    281 KBytes       
[  5]   9.00-10.00  sec  64.9 MBytes   544 Mbits/sec    0    281 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   653 MBytes   548 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   652 MBytes   547 Mbits/sec                  receiver

iperf Done.

Freebsd 13 Kernel on J1800 12.1 zfs installation (stock - 10222020 snapshot)

Code: [Select]

avery@debbox:~$ iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.39 port 42638 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  65.9 MBytes   552 Mbits/sec    0    185 KBytes       
[  5]   1.00-2.00   sec  64.9 MBytes   545 Mbits/sec    0    185 KBytes       
[  5]   2.00-3.00   sec  64.6 MBytes   542 Mbits/sec    0    185 KBytes       
[  5]   3.00-4.00   sec  64.7 MBytes   543 Mbits/sec    0    185 KBytes       
[  5]   4.00-5.00   sec  64.9 MBytes   544 Mbits/sec    0    185 KBytes       
[  5]   5.00-6.00   sec  64.9 MBytes   544 Mbits/sec    0    185 KBytes       
[  5]   6.00-7.00   sec  64.6 MBytes   542 Mbits/sec    0    185 KBytes       
[  5]   7.00-8.00   sec  65.0 MBytes   545 Mbits/sec    0    185 KBytes       
[  5]   8.00-9.00   sec  64.5 MBytes   541 Mbits/sec    0    185 KBytes       
[  5]   9.00-10.00  sec  64.9 MBytes   544 Mbits/sec    0    185 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   649 MBytes   544 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   648 MBytes   544 Mbits/sec                  receiver

iperf Done.
OPNsense 20.7.4 Kernel on J1800 12.1 zfs installation (https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/sets/kernel-20.7.4-amd64.txz)

Code: [Select]

iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.39 port 44048 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  64.6 MBytes   542 Mbits/sec    0    170 KBytes       
[  5]   1.00-2.00   sec  63.8 MBytes   535 Mbits/sec    0    170 KBytes       
[  5]   2.00-3.00   sec  63.1 MBytes   529 Mbits/sec    0    170 KBytes       
[  5]   3.00-4.00   sec  63.4 MBytes   532 Mbits/sec    0    170 KBytes       
[  5]   4.00-5.00   sec  63.8 MBytes   535 Mbits/sec    0    170 KBytes       
[  5]   5.00-6.00   sec  63.4 MBytes   532 Mbits/sec    0    180 KBytes       
[  5]   6.00-7.00   sec  63.8 MBytes   535 Mbits/sec    0    180 KBytes       
[  5]   7.00-8.00   sec  63.4 MBytes   532 Mbits/sec    0    180 KBytes       
[  5]   8.00-9.00   sec  63.8 MBytes   535 Mbits/sec    0    180 KBytes       
[  5]   9.00-10.00  sec  63.4 MBytes   532 Mbits/sec    0    180 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   636 MBytes   534 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   636 MBytes   533 Mbits/sec                  receiver

iperf Done.

Pretty much hair splitting, but interestingly the OPNsense 20.7.4 kernel is the slowest.  I wonder if this becomes more pronounced on faster systems.  I'll test it on my 10Gbe system later and report back.