Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - hax0rwax0r

Pages: [1]

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: September 03, 2020, 05:20:39 pm »

Quote from: mimugmail on September 03, 2020, 06:15:29 am

My first thought was maybe shared forwarding, but you have this with pfsense 2.5 too, correct?

I have never tested pfSense 2.5. As you had previously pointed out, my test was pfSense 2.4 which was FreeBSD 11.3 based. I mistakenly looked at the version history page and mentioned it was FreeBSD 12.1 but we determined I was incorrect in my statement.

Quote from: mimugmail on September 03, 2020, 12:26:17 pm

Ok, iflib, so it's related to 12.X-only, but strange it doesn't happen to vanilla 12.1

https://forums.freebsd.org/threads/what-is-kernel-if_io_tqg-100-load-of-core.70642/

Yeah, I saw that forum post when I was Googling around, too. I don't know what is different than vanilla FreeBSD 12.1 and the OPNsense 20.7.2 kernel that makes it higher CPU usage but it is consistent in my testing every single time.

Quote from: mimugmail on September 03, 2020, 12:36:34 pm

Do you still test with this hardware?
Dell T20 (Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (4 cores))

No, every single test, with the exception of that single test I did on the Dell T20 to see if more MHz helped, has been on a Dell R430. I have several R430 that are like-for-like and I even ran different software on each one and the results were consistent to weed out that a X520 NIC or something was bad. The results followed the OS/kernel installed regardless of which R430 I ran it on so I am fairly confident in my hardware.

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: September 02, 2020, 10:40:50 pm »

OK here are the test results as you requested:

FreeBSD 12.1 (pf enabled):

[root@fbsd1 ~]# uname -rv 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC [root@fbsd1 ~]# top -aSH last pid: 2954; load averages: 0.44, 0.42, 0.41 up 0+01:38:55 20:13:46 132 threads: 10 running, 104 sleeping, 18 waiting CPU: 0.0% user, 0.0% nice, 19.7% system, 5.2% interrupt, 75.1% idle Mem: 10M Active, 6100K Inact, 271M Wired, 21M Buf, 39G Free Swap: 3968M Total, 3968M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0 96K RUN 5 94:58 95.25% [idle{idle: cpu5}] 11 root 155 ki31 0 96K CPU1 1 93:26 83.69% [idle{idle: cpu1}] 11 root 155 ki31 0 96K RUN 0 94:44 73.68% [idle{idle: cpu0}] 11 root 155 ki31 0 96K CPU4 4 93:15 72.51% [idle{idle: cpu4}] 11 root 155 ki31 0 96K CPU3 3 93:36 64.80% [idle{idle: cpu3}] 11 root 155 ki31 0 96K RUN 2 92:55 62.29% [idle{idle: cpu2}] 0 root -76 - 0 480K CPU2 2 0:05 34.76% [kernel{if_io_tqg_2}] 0 root -76 - 0 480K CPU3 3 0:14 33.49% [kernel{if_io_tqg_3}] 12 root -52 - 0 304K CPU0 0 26:23 29.62% [intr{swi6: task queue}] 0 root -76 - 0 480K - 4 0:05 23.31% [kernel{if_io_tqg_4}] 0 root -76 - 0 480K - 0 0:05 12.31% [kernel{if_io_tqg_0}] 0 root -76 - 0 480K - 1 0:04 10.01% [kernel{if_io_tqg_1}] 12 root -88 - 0 304K WAIT 5 3:55 2.28% [intr{irq264: mfi0}] 0 root -76 - 0 480K - 5 0:06 1.88% [kernel{if_io_tqg_5}] 2954 root 20 0 13M 3676K CPU5 5 0:00 0.02% top -aSH 12 root -60 - 0 304K WAIT 0 0:01 0.01% [intr{swi4: clock (0)}] 0 root -76 - 0 480K - 4 0:02 0.01% [kernel{if_config_tqg_0}]

Single Thread:
[ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 8.45 GBytes 7.26 Gbits/sec 802 sender [ 4] 0.00-10.00 sec 8.45 GBytes 7.26 Gbits/sec receiver

10 Threads:
[ ID] Interval Transfer Bandwidth Retr [SUM] 0.00-10.00 sec 9.85 GBytes 8.46 Gbits/sec 2991 sender [SUM] 0.00-10.00 sec 9.83 GBytes 8.45 Gbits/sec receiver

FreeBSD 12.1 with OPNsense Kernel (pf enabled):

[root@fbsd1 ~]# uname -rv 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC [root@fbsd1 ~]# fetch https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/sets/kernel-20.7.2-amd64.txz [root@fbsd1 ~]# mv /boot/kernel /boot/kernel.old [root@fbsd1 ~]# tar -C / -xf kernel-20.7.2-amd64.txz [root@fbsd1 ~]# kldxref /boot/kernel [root@fbsd1 ~]# reboot [root@fbsd1 ~]# uname -rv 12.1-RELEASE-p8-HBSD FreeBSD 12.1-RELEASE-p8-HBSD #0 b3665671c4d(stable/20.7)-dirty: Thu Aug 27 05:58:53 CEST 2020 root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP [root@fbsd1 ~]# top -aSH last pid: 43891; load averages: 0.99, 0.49, 0.20 up 0+00:04:28 20:29:24 131 threads: 13 running, 100 sleeping, 18 waiting CPU: 0.0% user, 0.0% nice, 62.5% system, 3.5% interrupt, 33.9% idle Mem: 14M Active, 1184K Inact, 270M Wired, 21M Buf, 39G Free Swap: 3968M Total, 3968M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -76 - 0 480K CPU3 3 0:08 81.27% [kernel{if_io_tqg_3}] 0 root -76 - 0 480K CPU1 1 0:09 74.39% [kernel{if_io_tqg_1}] 0 root -76 - 0 480K CPU5 5 0:08 73.20% [kernel{if_io_tqg_5}] 0 root -76 - 0 480K CPU0 0 0:21 71.79% [kernel{if_io_tqg_0}] 11 root 155 ki31 0 96K RUN 4 4:09 54.15% [idle{idle: cpu4}] 11 root 155 ki31 0 96K RUN 2 4:09 51.30% [idle{idle: cpu2}] 0 root -76 - 0 480K CPU2 2 0:05 40.10% [kernel{if_io_tqg_2}] 0 root -76 - 0 480K - 4 0:09 37.60% [kernel{if_io_tqg_4}] 11 root 155 ki31 0 96K RUN 0 4:03 26.48% [idle{idle: cpu0}] 11 root 155 ki31 0 96K RUN 5 4:14 25.87% [idle{idle: cpu5}] 11 root 155 ki31 0 96K RUN 1 4:09 24.32% [idle{idle: cpu1}] 12 root -52 - 0 304K RUN 2 1:12 20.63% [intr{swi6: task queue}] 11 root 155 ki31 0 96K CPU3 3 4:00 17.30% [idle{idle: cpu3}] 12 root -88 - 0 304K WAIT 5 0:10 1.47% [intr{irq264: mfi0}] 43891 root 20 0 13M 3660K CPU4 4 0:00 0.03% top -aSH 21 root -16 - 0 16K - 4 0:00 0.02% [rand_harvestq] 12 root -60 - 0 304K WAIT 1 0:00 0.02% [intr{swi4: clock (0)}]

Single Thread:
[ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 2.89 GBytes 2.48 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 2.89 GBytes 2.48 Gbits/sec receiver

10 Threads:
[ ID] Interval Transfer Bandwidth Retr [SUM] 0.00-10.00 sec 8.16 GBytes 7.01 Gbits/sec 4260 sender [SUM] 0.00-10.00 sec 8.13 GBytes 6.98 Gbits/sec receiver

I included the "top -aSH" output again because my general observation between OPNsense kernel and FreeBSD 12.1 stock kernel is the "[kernel{if_io_tqg_X}]" process usage. Even on an actual OPNsense 20.7.2 installation I notice the exact same behavior of the "[kernel{if_io_tqg_X}]" being consistently higher and throughput significantly slower, specifically on single threaded tests. Note that both of the top outputs were only from the 10 thread count tests only as I did not think to capture them during the single threaded test.

I can't help but think that whatever high "[kernel{if_io_tqg_X}]" on the OPNsense kernel means is starving the system of throughput potential.

Thoughts? Next steps I can run and provide results from?

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: September 02, 2020, 07:25:58 pm »

Oh, I guess I misunderstood franco's instructions I thought they were asking me to drop the 20.7.2 kernel linked on top/in place on my FreeBSD 12.1 install which I was asking how exactly to do that.

I think with your clarification and re-reading the post, franco was just asking me to try an install of 20.7.2, which happens to be running that kernel, and re-run my tests to see if it improves.

If that's the case, I will try and report back my findings with OPNsense 20.7.2.

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: September 02, 2020, 05:12:22 pm »

I am not super familiar with FreeBSD so how would I go about swapping your kernel in for the existing stock FreeBSD 12.1 one I am running? I searched around on Google and I found how to build a customer kernel from source but this txz file you linked appears to be already compiled so I don't think that's what I want to do.

I also found reference to pkg-static to install locally downloaded packages but wanted to get some initial guidance before totally hosing this up.

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: September 02, 2020, 07:34:01 am »

OK, back to basics here. I couldn't leave well enough alone and I did more testing tonight because I just couldn't believe that my CPU couldn't even do single threaded gigabit. Here's my test scenario:

Test Scenario 1:

Physical Linux Server (CentOS 7) on VLAN 2 (iperf3 client)
Virtual Linux Server (CentOS 7) on VLAN 24 (iperf3 server)
Dell PowerEdge R430 w/Intel X520-SR2 and HardenedBSD 12-STABLE (BUILD-LATEST 2020-08-31)

Single Threaded:
[ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.00 GBytes 863 Mbits/sec 0 sender [ 4] 0.00-10.00 sec 1.00 GBytes 860 Mbits/sec receiver

6 Parallel Threads:
[ ID] Interval Transfer Bandwidth Retr [SUM] 0.00-10.00 sec 2.23 GBytes 1.91 Gbits/sec 938 sender [SUM] 0.00-10.00 sec 2.22 GBytes 1.90 Gbits/sec receiver

Notice a common theme here with the ~850 Mbps single threaded test. It's pretty close to what I get with OPNsense. Note this is THROUGH the firewall and not from the firewall. Also note my system did have IPv6 addresses from my ISP on each of the interfaces, though, I was only testing IPv4 traffic.

Test Scenario 2:

Physical Linux Server (CentOS 7) on VLAN 2 (iperf3 client)
Virtual Linux Server (CentOS 7) on VLAN 24 (iperf3 server)
Dell PowerEdge R430 w/Intel X520-SR2 and FreeBSD 12.1-RELEASE

Single Threaded:
[ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 9.75 GBytes 8.38 Gbits/sec 573 sender [ 4] 0.00-10.00 sec 9.75 GBytes 8.38 Gbits/sec receiver

6 Parallel Threads:
[ ID] Interval Transfer Bandwidth Retr [SUM] 0.00-10.00 sec 10.5 GBytes 9.05 Gbits/sec 3607 sender [SUM] 0.00-10.00 sec 10.5 GBytes 9.04 Gbits/sec receiver

I couldn't believe my eyes as I had to do a triple check that it was in fact pushing 8.38 Gbps THROUGH the FreeBSD 12.1 server and it wasn't taking some magical alternate path somehow. It was, in fact, going through the FreeBSD router. As you can see, parallel test is about 1 Gbps less than wire speed. Excellent! Also note my system did have IPv6 addresses from my ISP on each of the interfaces, though, I was only testing IPv4 traffic.

I thought I would enable pfctl on the FreeBSD 12.1 router to see how that affected performance. Not sure how much adding rules impacts throughput but I did notice a measurable drop in the single thread test (6.23 Gbps) but the parallel thread test was negligible (8.94 Gbps).

As of right now, it seems so so so strange to me that HardenedBSD exhibits the same exact single threaded throughput and likewise low parallel thread throughput over FreeBSD.

I am willing to accept that I am not accounting for something here; however, near wire speed throughput on the same exact hardware on FreeBSD versus HardenedBSD, it seems to me something is very different with HardenedBSD.

What are your thoughts?

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: August 31, 2020, 08:17:06 am »

I built a new OPNsense server on my spare Dell PowerEdge R430 server that has the same CPU in it as my one I am currently using.

I can confirm that the problem appears to be my CPU and/or hardware since the same exact NIC was moved from the Dell PowerEdge T20 which previously tested out at 7.53 Gbps to this R430 server and the test results are much lower:

[root@client1 ~]# iperf3 -c 192.168.1.31 ... [ 4] 0.00-10.00 sec 2.13 GBytes 1.83 Gbits/sec 72 sender [ 4] 0.00-10.00 sec 2.13 GBytes 1.83 Gbits/sec receiver [root@client1 ~]# iperf3 -c 192.168.1.31 -P 10 ... [SUM] 0.00-10.00 sec 4.78 GBytes 4.10 Gbits/sec 1143 sender [SUM] 0.00-10.00 sec 4.75 GBytes 4.08 Gbits/sec receiver

One observation is on like-for-like hardware, the new R430 is performing more than double the throughput on the single thread test and more than a gigabit more on parallel test than my currently used R430 I have been experiencing problems with. No idea why this is.

I guess I have a decision to make about buying a new CPU or a new server.

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: August 28, 2020, 09:08:12 am »

Fresh install of OPNsense 20.7 on a Dell T20 (Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (4 cores)):

[root@client1 ~]# iperf3 -c 192.168.1.31 ... [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 8.29 GBytes 7.12 Gbits/sec 2 sender [ 4] 0.00-10.00 sec 8.29 GBytes 7.12 Gbits/sec receiver

[root@client1 ~]# iperf3 -c 192.168.1.31 -P 10 ... [ ID] Interval Transfer Bandwidth Retr [SUM] 0.00-10.01 sec 8.77 GBytes 7.53 Gbits/sec 139 sender [SUM] 0.00-10.01 sec 8.77 GBytes 7.53 Gbits/sec receiver

It's just hard to believe that E3-1225 v3 @ 2.4GHz/3.2GHz versus an E5-2620 v3 3.2GHz/3.6GHz is that much difference for a single thread test; however, it's clear, the results don't lie. There's either something wrong with my hardware, my install or it's just too slow of a CPU to push single threaded performance past about 850 Mbps.

And you're right about the pfSense version of FreeBSD. I just double checked the page (https://docs.netgate.com/pfsense/en/latest/releases/versions-of-pfsense-and-freebsd.html) and, in spite of it being clearly marked 2.5.0 TBD, I didn't even pay attention that it definitely was not the edition I installed.

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: August 28, 2020, 07:45:31 am »

I mean of course running a parallel test is going to yield better results if the firewall has multi-core CPU(s) and you are maxing out a CPU core.

The issue I have is that that single threaded throughput is only about 850 Mbps on my non-virtualized hardware. That seems not right to me but I only know my environment so I might just be wrong.

And yes, I did test through the firewall before I started doing tests from the firewall. Through the firewall nets me similar performance for single threaded:

[root@client1 ~]# iperf3 -f m -c 192.168.1.31 ... [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 973 MBytes 816 Mbits/sec 22 sender [ 4] 0.00-10.00 sec 970 MBytes 814 Mbits/sec receiver

And, as expected, increased throughput when running in parallel:

[root@client1 ~]# iperf3 -f m -c 192.168.1.31 -P 10 ... [ ID] Interval Transfer Bandwidth Retr ... [SUM] 0.00-10.00 sec 3.26 GBytes 2798 Mbits/sec 4464 sender [SUM] 0.00-10.00 sec 3.23 GBytes 2776 Mbits/sec receiver

~~Can you humor me and run a single threaded test through your hardware and show me the output?~~

If OPNsense is truly not broken in this release then I guess my CPU core speed isn't enough to achieve what I am looking to do and I need to look on eBay for a faster one. That being said, it appears there are several others reporting degraded performance since upgrading so maybe there is something to my claim.

Edit: I see your single threaded non-IPS throughput is 6826 Mbps. See, even your single threaded test absolutely crushes mine. I get that your CPU is @ 3.7 GHz and a v6 but really, almost 7 Gbps versus less than my 1 Gbps. I have a v3 Xeon that has a higher clock rate (maybe 3.2 GHz?) I can try to test out tomorrow to see what results I get.

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: August 28, 2020, 04:08:35 am »

OK so at the risk of seeming like I am only talking to myself at this point, I think I found a commonality amongst the poor performance -- it's OPNsense.

I built a fresh new and updated OPNsense 20.7.1 VM on VMware ESXi 6.7U3, imported my configuration backup from my physical server and re-mapped all the interfaces to the new vmx0_vlanX names and things are working, albeit even slower than the physical hardware:

root@opnsense1:~ # iperf3 -c 192.168.1.31 ... [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 705 MBytes 591 Mbits/sec 0 sender [ 5] 0.00-10.41 sec 705 MBytes 568 Mbits/sec receiver

Seems pretty awful. So I decided to create a two new OPNsense 20.7.1 VMs and configure one as a VLAN trunk and the other as non-trunk to test if the problem lied within the VLAN implementation itself:

OPNsense 20.7.1 (amd64)

VLAN and pf Enabled:
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 949 MBytes 796 Mbits/sec 0 sender [ 5] 0.00-10.40 sec 949 MBytes 766 Mbits/sec receiver

VLAN and pf Disabled (pfctl -d):
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 1.22 GBytes 1.05 Gbits/sec 0 sender [ 5] 0.00-10.41 sec 1.22 GBytes 1.01 Gbits/sec receiver

Non-VLAN and pf Enabled:
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 854 MBytes 716 Mbits/sec 0 sender [ 5] 0.00-10.40 sec 854 MBytes 688 Mbits/sec receiver

Non-VLAN and pf Disabled (pfctl -d):
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 983 MBytes 825 Mbits/sec 0 sender [ 5] 0.00-10.40 sec 983 MBytes 793 Mbits/sec receiver

As you can see, the VLAN trunk configured VM had slightly better performance. Perhaps environmental impacts caused the performance differences as I would expect them to be nearly the same. Even at the differences I'm seeing, I would consider it mostly negligible given the link is 10 gigabit. I also tested without pf to see if the throughput was measurable. Both tests show that it is in fact better without pf, though, kinda pointless to have a network perimeter firewall without it running...

Next I thought maybe this is just a fluke and all three OPNsense servers just suck on VMware ESXi and dislike the hardware or configuration or maybe my ESX host just can't push traffic. I had a CentOS 8.2.2004 VM already deployed and configured on the same network segment I had been testing on so I loaded up iperf3 on it to see if it was an ESX host/network problem.

CentOS 8.2.2004 (x86_64)

Non-VLAN and firewalld Enabled:
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.7 GBytes 9.17 Gbits/sec 11 sender [ 5] 0.00-10.04 sec 10.7 GBytes 9.14 Gbits/sec receiver
Non-VLAN and firewalld Disabled:
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.8 GBytes 9.32 Gbits/sec 1 sender [ 5] 0.00-10.04 sec 10.8 GBytes 9.28 Gbits/sec receiver

Tested with firewall on and off just for fun to see how much iptables slowed the Linux test down. As you can see, 9.14 Gbps to 9.32 Gbps on this test. The problem isn't my ESX host or my network.

I then thought it might be a BSD problem. Perhaps something with running inside VMware or the vmxnet3 driver that is problematic. I tried to figure out how to install HardenedBSD but it seemed too difficult difficult as my quick search for an ISO yielded not much. As such, I used FreeBSD. Hopefully it's close enough!

FreeBSD 12.1 (amd64)

VLAN and pf Disabled (not configured):
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.35 Gbits/sec 0 sender [ 5] 0.00-10.42 sec 10.9 GBytes 8.97 Gbits/sec receiver

Non-VLAN and pf Disabled (not configured):
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.36 Gbits/sec 13 sender [ 5] 0.00-10.21 sec 10.9 GBytes 9.17 Gbits/sec receiver

I thought I hadn't spent enough time already dorking around with this so why not configure one test VM to be VLAN trunking and the other not to see if there are any differences. As you can see, FreeBSD 12.1 pushed the packets, fast, regardless of VLAN or otherwise. Problem doesn't seem to be vmxnet3/ESXi and FreeBSD related.

Finally, I came to the conclusion that maybe OPNsense 20.7 is just broken. As such, I loaded up a OPNsense 19.7 test VM and gave it a go.

OPNsense 19.7.10_1 (amd64)

Non-VLAN and pf Enabled:
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.75 GBytes 1.50 Gbits/sec 0 sender [ 5] 0.00-10.44 sec 1.75 GBytes 1.44 Gbits/sec receiver
Non-VLAN and pf Disabled (pfctl -d):
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 2.57 GBytes 2.21 Gbits/sec 0 sender [ 5] 0.00-10.48 sec 2.57 GBytes 2.11 Gbits/sec receiver

Not good. You can see the results of 1.75 Gbps to 2.57 Gbps is measurably better than my test results with OPNsense 20.7 but nowhere near stellar. I was very much over testing at this point so I opted not to do a VLAN versus non-VLAN configuration. That being said, based on historical results, I am sure that the difference in results would have been negligible.

To add to this, as a general observation, whenever the iperf3 test is running on OPNsense, a constant ping of the firewall starts to drop packets like it is choked out and cannot keep up. I did not experience this at all on CentOS or FreeBSD when testing.

Why is OPNsense so bad at throughput in my tests? If it's not, what am I doing wrong? The commonality amongst these tests seems to be OPNsense, regardless if it's 19.7 or 20.7, though, the former is better than the later.

Edit: Because why not at this point. Let's test pfSense!

pfSense 2.4.5 (amd64)

Non-VLAN and pf Enabled:
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 3.80 GBytes 3.26 Gbits/sec 67 sender [ 5] 0.00-10.26 sec 3.80 GBytes 3.18 Gbits/sec receiver

Non-VLAN and pf Disabled (pfctl -d):
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 5.66 GBytes 4.86 Gbits/sec 109 sender [ 5] 0.00-10.22 sec 5.66 GBytes 4.76 Gbits/sec receiver

pfSense is not stellar, especially considering it is based on FreeBSD 12.1 and I tested FreeBSD 12.1 and got very different (better) results. That being said, both results are much, much faster than any OPNsense test I could push regardless if physical or virtual.

Edit 2: Fixed a typo in my comments where I erroneously used 20.1 instead of 20.7 when referring to editions of OPNsense.

TL;DR: OPNsense seems to be dog slow compared to FreeBSD 12.1 and CentOS 8.2 at raw network throughput. What gives? What am I doing wrong that it can be this huge of a performance gap?

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: August 27, 2020, 03:41:43 am »

Just a status update:

Swapped optics on the switch side (both have now been switched) and swapped for a new fiber patch cable. Same results. I also re-enabled "Hardware CRC" and "VLAN Hardware Filtering" but left "Hardware TSO" and "Hardware LRO" disabled as I read most drivers are broken for those functions.

I also added this to /boot/loader.conf.local and rebooted:

hw.cxgbe.toecaps_allowed=0 hw.cxgbe.rdmacaps_allowed=0 hw.cxgbe.iscsicaps_allowed=0 hw.cxgbe.fcoecaps_allowed=0

Absolutely zero impact in performance. Tomorrow I think I'll unbox my other PowerEdge R430 and put the original Intel X520-SR2 NIC in it and see if I can duplicate the problem.

I am at a total loss of what is going on here.

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: August 26, 2020, 09:42:16 am »

I know that the Broadcom drivers aren't the best but I figured it was worth a test. That being said, I just swapped the Intel X520-SR2 with a Chelsio T540-CR which seems to have excellent FreeBSD support and that family of NICs seems frequently recommended.

Here's the results from the Chelsio T540-CR:

# iperf3 -c 192.168.1.31 Connecting to host 192.168.1.31, port 5201 [ 5] local 192.168.1.1 port 19465 connected to 192.168.1.31 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 112 MBytes 943 Mbits/sec 0 8.00 MBytes [ 5] 1.00-2.00 sec 110 MBytes 924 Mbits/sec 0 8.00 MBytes [ 5] 2.00-3.00 sec 112 MBytes 939 Mbits/sec 0 8.00 MBytes [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 0 8.00 MBytes [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 8.00 MBytes [ 5] 5.00-6.00 sec 112 MBytes 939 Mbits/sec 0 8.00 MBytes [ 5] 6.00-7.00 sec 112 MBytes 940 Mbits/sec 0 8.00 MBytes [ 5] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 0 8.00 MBytes [ 5] 8.00-9.00 sec 112 MBytes 940 Mbits/sec 0 8.00 MBytes [ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec 0 8.00 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec 0 sender [ 5] 0.00-10.32 sec 1.09 GBytes 909 Mbits/sec receiver

Also thought it was interesting there were zero retransmits on the test.

I swapped out the optic on the NIC when I swapped the NIC itself. I will swap the optic on the switch and maybe try a different switch port and fiber patch cable tomorrow, though, I doubt those are the issue.

Unfortunately, it appears that the issue was not my Intel X520-SR2 NIC as the Chelsio T540-CR exhibits the same behavior.

Hardware and Performance / Re: Poor Throughput (Even On Same Network Segment)

« on: August 26, 2020, 08:17:40 am »

To add to this, I re-configured all my VLANs on bge0 (onboard NIC) and moved all my interfaces over to each respective bge0_vlanX interface and re-ran my iperf3 tests.

On my first test, I got the same throughput as with my Intel X520-SR2 NIC:

# iperf3 -c 192.168.1.31 Connecting to host 192.168.1.31, port 5201 [ 5] local 192.168.1.1 port 42455 connected to 192.168.1.31 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 92.0 MBytes 772 Mbits/sec 91 5.70 KBytes [ 5] 1.00-2.00 sec 91.1 MBytes 764 Mbits/sec 88 145 KBytes [ 5] 2.00-3.00 sec 86.1 MBytes 722 Mbits/sec 86 836 KBytes [ 5] 3.00-4.00 sec 92.5 MBytes 776 Mbits/sec 76 589 KBytes [ 5] 4.00-5.00 sec 107 MBytes 894 Mbits/sec 0 803 KBytes [ 5] 5.00-6.00 sec 107 MBytes 898 Mbits/sec 2 731 KBytes [ 5] 6.00-7.00 sec 109 MBytes 914 Mbits/sec 1 658 KBytes [ 5] 7.00-8.00 sec 110 MBytes 926 Mbits/sec 0 863 KBytes [ 5] 8.00-9.00 sec 107 MBytes 898 Mbits/sec 2 748 KBytes [ 5] 9.00-10.00 sec 109 MBytes 918 Mbits/sec 1 663 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1011 MBytes 848 Mbits/sec 347 sender [ 5] 0.00-10.32 sec 1010 MBytes 821 Mbits/sec receiver

For reference, I just tested with my MacBook Pro against the same iperf3 server and was able to push 926 Mbps and re-tested my QNAP to QNAP transfer and it did 9.39 Gbps to completely rule out it's an iperf3 server thing.

For the sake of testing because why not, I re-ran iperf3 from my OPNsense server once more and got near gigabit throughput:

# iperf3 -c 192.168.1.31 Connecting to host 192.168.1.31, port 5201 [ 5] local 192.168.1.1 port 8283 connected to 192.168.1.31 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 108 MBytes 906 Mbits/sec 0 792 KBytes [ 5] 1.00-2.00 sec 111 MBytes 932 Mbits/sec 2 698 KBytes [ 5] 2.00-3.00 sec 111 MBytes 930 Mbits/sec 1 638 KBytes [ 5] 3.00-4.00 sec 108 MBytes 905 Mbits/sec 1 585 KBytes [ 5] 4.00-5.00 sec 111 MBytes 929 Mbits/sec 0 816 KBytes [ 5] 5.00-6.00 sec 111 MBytes 929 Mbits/sec 1 776 KBytes [ 5] 6.00-7.00 sec 111 MBytes 928 Mbits/sec 1 725 KBytes [ 5] 7.00-8.00 sec 108 MBytes 906 Mbits/sec 2 663 KBytes [ 5] 8.00-9.00 sec 111 MBytes 928 Mbits/sec 2 616 KBytes [ 5] 9.00-10.00 sec 111 MBytes 928 Mbits/sec 0 837 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.07 GBytes 922 Mbits/sec 10 sender [ 5] 0.00-10.32 sec 1.07 GBytes 892 Mbits/sec receiver

One thing I noticed between the first and second iperf3 test was the "Retr" column of 347 vs 10. I researched what that meant for iperf3 and found this: "It's the number of TCP segments retransmitted. This can happen if TCP segments are lost in the network due to congestion or corruption."

I also noticed during my second iperf3 test that there was now a kernel process using 99.81% CPU:

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0 192K CPU3 3 9:02 100.00% [idle{idle: cpu3}] 0 root -92 - 0 848K CPU2 2 0:30 99.81% [kernel{bge0 taskq}]

Additionally, I am not sure "Retr" in itself is a smoking gun as the QNAP to QNAP test that yielded 9.39 Gbps did 2218 retries.

The search continues.

Hardware and Performance / Poor Throughput (Even On Same Network Segment)

« on: August 25, 2020, 08:31:25 pm »

I originally posted on Reddit but figured I might get more traction here with this.

I have an OPNsense 20.7.1 server running on a Dell R430 with 16 GB DDR4 RAM, an Intel Xeon E5-2620 v3 (6 cores/12 threads @ 2.40GHz) CPU and an Intel X520-SR2 10GbE NIC.

My network has several VLANs and network subnets with my OPNsense router functioning as a router on a stick doing all the traffic firewalling and routing between each network segment.

I recently upgraded my OPNsense to 20.7.1 and on a whim decided to run an iperf3 test between two VMs on different network segments to see what kind of throughput I was getting. I am certain, at least at some point, this very same hardware pushed over 6 Gbps on the same iperf3 test. Today it was getting around 850 Mbps every single time.

I started iperf3 as a server on my QNAP NAS device which is also attached to the same 10 Gbps switch and ran iperf3 as a client from OPNsense on the same network segment and got the same 850 Mbps throughput.

To make sure I wasn't limited by the QNAP NAS device, I ran the same iperf3 test with my other QNAP NAS device as a client to the first QNAP NAS device and it pushed 8.6 Gbps across the same network segment (no OPNsense involved) so both the QNAP and the switch can push it.

My question is what do I have going wrong here? Even the same network segment, OPNsense can't do more than 850 Mbps throughput. I have no idea if this was happening pre-upgrade to 20.7.1 but I know for sure it is happening now. I would assume an iperf3 test from the OPNsense server on the same network segment would surely remove any doubt it was firewalling, etc.

The interface shows 10 Gbps link speed, too, both from ifconfig and the switch itself.

My current MBUF Usage is 1 % (17726/1010734).

IDS/IPS package is installed but disabled.

I had "Hardware CRC" and "Hardware TSO" and "Hardware LRO" and "VLAN Hardware Filtering" all enabled. I have since set those all to disabled and rebooted. I can confirm that it disabled by looking at the interface flags in ifconfig:

Pre-reboot:
options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
Post-reboot:
options=803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
I ran top and was able to see a process (kernel{if_io_tqg_2}) utilize near 100% of a CPU core during this iperf3 test:

# top -aSH last pid: 22772; load averages: 1.23, 0.94, 0.79 up 5+23:48:52 14:24:22 233 threads: 15 running, 193 sleeping, 25 waiting CPU: 1.0% user, 0.0% nice, 16.1% system, 0.5% interrupt, 82.4% idle Mem: 1485M Active, 297M Inact, 1657M Wired, 935M Buf, 12G Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -76 - 0 848K CPU2 2 279:51 99.77% [kernel{if_io_tqg_2}] 11 root 155 ki31 0 192K CPU3 3 130.8H 98.78% [idle{idle: cpu3}] 11 root 155 ki31 0 192K CPU9 9 131.3H 98.75% [idle{idle: cpu9}] 11 root 155 ki31 0 192K CPU1 1 129.7H 98.68% [idle{idle: cpu1}] 11 root 155 ki31 0 192K CPU10 10 138.1H 98.33% [idle{idle: cpu10}] 11 root 155 ki31 0 192K CPU5 5 130.5H 97.51% [idle{idle: cpu5}] 11 root 155 ki31 0 192K CPU0 0 138.3H 95.78% [idle{idle: cpu0}] 11 root 155 ki31 0 192K CPU8 8 137.7H 95.25% [idle{idle: cpu8}] 11 root 155 ki31 0 192K CPU6 6 138.7H 95.20% [idle{idle: cpu6}] 11 root 155 ki31 0 192K CPU4 4 138.4H 94.26% [idle{idle: cpu4}] 22772 root 82 0 15M 6772K CPU7 7 0:04 93.83% iperf3 -c 192.168.1.31 11 root 155 ki31 0 192K RUN 7 129.4H 68.75% [idle{idle: cpu7}] 11 root 155 ki31 0 192K RUN 11 126.8H 46.12% [idle{idle: cpu11}] 0 root -76 - 0 848K - 4 277:00 5.12% [kernel{if_io_tqg_4}] 12 root -60 - 0 400K WAIT 11 449:21 5.02% [intr{swi4: clock (0)}] 0 root -76 - 0 848K - 8 317:40 3.81% [kernel{if_io_tqg_8}] 0 root -76 - 0 848K - 0 272:13 2.71% [kernel{if_io_tqg_0}]
I occasionally see flowd_aggregate.py pop up to 100% but it doesn't seem consistent or relevant to when iperf3 is running:

# top -aSH last pid: 99781; load averages: 1.15, 0.90, 0.77 up 5+23:47:27 14:22:57 232 threads: 14 running, 193 sleeping, 25 waiting CPU: 8.5% user, 0.0% nice, 1.6% system, 0.4% interrupt, 89.5% idle Mem: 1481M Active, 299M Inact, 1656M Wired, 935M Buf, 12G Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 43465 root 90 0 33M 25M CPU7 7 7:11 99.82% /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.7) 11 root 155 ki31 0 192K CPU9 9 131.3H 99.80% [idle{idle: cpu9}] 11 root 155 ki31 0 192K CPU3 3 130.8H 99.68% [idle{idle: cpu3}] 11 root 155 ki31 0 192K CPU10 10 138.1H 99.50% [idle{idle: cpu10}] 11 root 155 ki31 0 192K CPU6 6 138.7H 98.53% [idle{idle: cpu6}] 11 root 155 ki31 0 192K RUN 5 130.5H 98.20% [idle{idle: cpu5}] 11 root 155 ki31 0 192K CPU1 1 129.7H 97.97% [idle{idle: cpu1}] 11 root 155 ki31 0 192K CPU11 11 126.8H 96.52% [idle{idle: cpu11}] 11 root 155 ki31 0 192K CPU0 0 138.3H 96.43% [idle{idle: cpu0}] 11 root 155 ki31 0 192K CPU8 8 137.7H 95.95% [idle{idle: cpu8}] 11 root 155 ki31 0 192K CPU2 2 138.3H 95.81% [idle{idle: cpu2}] 11 root 155 ki31 0 192K CPU4 4 138.4H 93.94% [idle{idle: cpu4}] 12 root -60 - 0 400K WAIT 4 449:17 5.10% [intr{swi4: clock (0)}] 0 root -76 - 0 848K - 4 276:55 4.95% [kernel{if_io_tqg_4}]

What is going on here?

Pages: [1]