OPNsense Forum
Archive => 22.1 Legacy Series => Topic started by: NW4FUN on March 26, 2022, 10:23:10 am
-
Hello,
I'm fairly new to OPNsense as it has been sitting in my home lab for just a few months. I've had the 22.1.x community version running on 2 bare metals and it was all working just fine to the point I've decided to buy an "official appliance" - DEC3840 - for my prod environment.
This is where things started going south...
In a nutshell, this is running on 21.10.3 Business edition (I've noticed this is based on FreeBSD12 rather than 13) and my WAN pppoe throughput reaches ~300M at its best. I have a symmetric 10G line which was just working at full speed with the previous - supposedly lowered spec - HW.
I supposed I must have done something wrong? (I manually copied and pasted all of the previous settings and double/triple checked to make sure nothing was being forgotten)
Anyone might help me in troubleshooting this as I start thinking I might have received a faulty unit from Deciso...
Thanks for your support,
NW4FUN
-
I think the Deciso hardware is really fine but install the community edition, anyway. In your situation I would try that first. While you are at it you can install with ZFS.
-
Hey,
Thanks for your reply.
Why installing community edition and migrating to ZFS?
Can you walk me through the advantages compared to business edition on UFS.
Cheers,
NW4FUN
-
ZFS is way more resilient to unexpected outages. It's 2022 - I run ZFS literally everywhere. With the exception of virtual machines, because the copy-on-write nature of ZFS thwarts storage overprovisioning.
As for the community edition - get new features faster. E.g. the FreeBSD 13 kernel and vastly improved bridging performance among other things.
HTH,
Patrick
-
Thanks Bud,
Much appreciated. I’ve upgraded to 22.1.4_1 following your suggestion, however:
1) I wasn’t able to run a fresh install as it kept freezing at the same point (the only image able to be launched was SERIAL as the deciso appliance lacks of video output). Therefore couldn’t switch to ZFS (any suggestion appreciated)
2) the throughput hasn’t improved at as Speedtest from LAN sits at ~300M max while ran from cli sits at 1.9G
3) I can’t perform iPerf testing as client times out
4) cpu goes up to 100% very often without really doing anything
I’m really disappointed with this appliance so far, I keep thinking I’ve got a faulty unit and thinking of returning it.
My previous HW was running on i7-9700 with 16G Ram and it was flying. I was expecting at the very least same performances with the deciso 3840. Not the case.
I wish I could be able to run a clean install to rule out any potential misconfiguration or anything.
-
1) I wasn’t able to run a fresh install as it kept freezing at the same point (the only image able to be launched was
I had similar issues once, The issue was related to the sense image and or the usb stick...So i downloaded the image again, verified it + used a different usb stick !
-
Hey,
Thanks for your suggestion. I've tried 3 different USB dongles, always getting stuck at the same point.
Below what I get from the console (I guess tomorrow I'm going to initiate return, I'm insanely disappointed!!!)
Perhaps any of you guys have any idea on what I might be doing wrong??
Cheers
NW4FUN
Last login: Sun Mar 27 15:05:45 on ttys001
umass0: SCSI over Bulk-Only; quirks = 0x8100
umass0:1:0: Attached to scbus1
Root mount waiting for: usbus0 CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <TS256GMTS952T2 02J0T4GB> ACS-2 ATA SATA 3.x device
ada0: Serial Number G821760019
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 244198MB (500118192 512 byte sectors)
da0 at umass-sim0 bus 0 scbus1 target 0 lun 0
da0: <TDKMedia TF 150 Drive PMAP> Removable Direct Access SCSI device
da0: Serial Number 07B3080795C7178F
da0: 40.000MB/s transfers
da0: 3812MB (7806976 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
GEOM: da0: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-07B3080795C7178F: the secondary GPT header is not in the last LBA.
mountroot: waiting for device /dev/ufs/OPNsense_Install...
GEOM: diskid/DISK-07B3080795C7178F: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-07B3080795C7178F: the secondary GPT header is not in the last LBA.
GEOM: diskid/DISK-07B3080795C7178F: the secondary GPT header is not in the last LBA.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
intsmb0: <AMD FCH SMBus Controller> at device 20.0 on pci0
smbus0: <System Management Bus> on intsmb0
lo0: link state changed to UP
pflog0: permanently promiscuous mode enabled
-
You are using the serial image?
-
Should be serial already, the bigger question is if it was a 22.1.2 image or not (22.1 is buggy with FreeBSD 13) and on the device BIOS you need to disable legacy UART. It's all rather basic information that the existing customer support channel can also offer. :)
Cheers,
Franco
-
@franco you're spot on there!
Following the super caring advise of Deciso's support, I've disabled legacy UART in the BIOS and managed to run a fresh install. At least I've got that one down...
I'm still trying to get to the bottom of the very bizarre performances behaviour my machine is having.
I'm going to post a detailed report once I've finished troubleshooting so the community may benefit from my experience.
-
alright....as promised, after a very long period of testing, troubleshooting, network changes, etc... I've found a configuration that is consistently solid in terms of performances.
Apparently, having WAN configured on igb(x) and LAN on ax(x) delivers very satisfactory performances in terms of throughput. The drawback is with the above config I limit my link to just 1G instead of 10G.
What I've found interesting tho (already informed the DECISO's guys) is that everything breaks if either:
1. WAN goes over ax0/1
OR
2. LAN goes over LACP ax0,ax1
Cheers
-
alright....as promised, after a very long period of testing, troubleshooting, network changes, etc... I've found a configuration that is consistently solid in terms of performances.
Apparently, having WAN configured on igb(x) and LAN on ax(x) delivers very satisfactory performances in terms of throughput. The drawback is with the above config I limit my link to just 1G instead of 10G.
What I've found interesting tho (already informed the DECISO's guys) is that everything breaks if either:
1. WAN goes over ax0/1
OR
2. LAN goes over LACP ax0,ax1
Cheers
I'm also on a DEC3840 and I've been going through the same issues with you, apparently. For your console issue, you need to disable Legacy UART in the BIOS https://docs.opnsense.org/hardware/serial_connectivity.html#legacy-uart-vs-uefi-serial. That solved my console output issues.
I have multi-gig WAN as well and having WAN on ax1 and LAN on ax0 causes poor routing performance for me, I can't seem to get past 2-3Gbps when traffic comes *into* the box first. iperf tests from an interface across VLANs is always maxing out 10Gb.
Seems to maybe be an issue if both ax0 and ax1 are in use?
-
Interesting. I always thought this box wasn’t fit for purpose and tried to explain that to deciso.
I wonder whether the CPU is the culprit here?
-
Interesting. I always thought this box wasn’t fit for purpose and tried to explain that to deciso.
I wonder whether the CPU is the culprit here?
I don't think so, their tests clearly show >=10Gb throughput through the firewall. And doing iPerf from one VLAN to a device on a different VLAN from the opnsense CLI shows full 10Gb speed, so clearly the power is there, it's just when the traffic is coming into the box first is where I have performance issues.
[ 5] local 10.0.30.1 port 16402 connected to 10.0.70.4 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.07 GBytes 9.17 Gbits/sec 0 1.82 MBytes
[ 5] 1.00-2.00 sec 1.08 GBytes 9.24 Gbits/sec 0 2.53 MBytes
[ 5] 2.00-3.00 sec 1.09 GBytes 9.38 Gbits/sec 0 3.00 MBytes
[ 5] 3.00-3.64 sec 701 MBytes 9.23 Gbits/sec 0 3.00 MBytes
-
How are you measuring that?
IPerf plug-in on FW or IPerf.server-DEC3840-IPerf.client?
-
How are you measuring that?
IPerf plug-in on FW or IPerf.server-DEC3840-IPerf.client?
I believe just the iperf plugin on the firewall.
-
I see...
Directly plugged into DEC3840 or via Switch (if so, can you name make, model and SW version pls)?
-
* pppoe wan
* 10g
In my experience that did not work well with high bandwith, no matter if FreeBSD or Linux, pppoe has way to much overhead for that.
-
* pppoe wan
* 10g
In my experience that did not work well with high bandwith, no matter if FreeBSD or Linux, pppoe has way to much overhead for that.
I don't disagree with that, however, as part of my testing, I've had my ISP to provide me with DHCP instead of PPPoE and guess what? Situation remains unchanged...
-
I see...
Directly plugged into DEC3840 or via Switch (if so, can you name make, model and SW version pls)?
I'm actually going through multiple switches, mostly unifi equipment, Enterprise-8-PoE then a Switch Aggregation, the up to the DEC.
Latest versions on everything.
* pppoe wan
* 10g
In my experience that did not work well with high bandwith, no matter if FreeBSD or Linux, pppoe has way to much overhead for that.
* pppoe wan
* 10g
In my experience that did not work well with high bandwith, no matter if FreeBSD or Linux, pppoe has way to much overhead for that.
I don't disagree with that, however, as part of my testing, I've had my ISP to provide me with DHCP instead of PPPoE and guess what? Situation remains unchanged...
In my case, I'm not even using PPPoE and this is just within different VLAN networks.
With some tuning, I was able to get it up to about 7Gbps, but still much lower than the expected 14+ advertised.
-
Can you what tuning have you applied to increase performances?
-
Can you what tuning have you applied to increase performances?
Turn on VLAN Hardware Offload, enable the parent interface (no IPs or anything, just assign it and enable), also enabled HT interrupts as per https://wiki.freebsd.org/Networking/10GbE/Router#Allow_interrupts_on_HTT_logical_CPUs this BSD 10Gb tuning article
-
Turn on VLAN Hardware Offload, enable the parent interface (no IPs or anything, just assign it and enable), also enabled HT interrupts as per https://wiki.freebsd.org/Networking/10GbE/Router#Allow_interrupts_on_HTT_logical_CPUs this BSD 10Gb tuning article
I'm not sure how to enable HTT interrupts, however, would it make any sense at all since DEC3840 is AMD based whilst HT instructions are Intel's?
-
Turn on VLAN Hardware Offload, enable the parent interface (no IPs or anything, just assign it and enable), also enabled HT interrupts as per https://wiki.freebsd.org/Networking/10GbE/Router#Allow_interrupts_on_HTT_logical_CPUs this BSD 10Gb tuning article
I'm not sure how to enable HTT interrupts, however, would it make any sense at all since DEC3840 is AMD based whilst HT instructions are Intel's?
I assumed it may have had something to do with general multi-core enhancement, rather than *specifically* HT, I did see about a 1Gbps performance increase adding that, so /shrug.
-
Looks like 22.1.8 was just released, anyone update and see if there's any improvements?
-
I don't think there are significant changes on 22.1.x and I'm not sure why people sometimes wonder if there are... release notes state what is being worked on and source repositories have annotated changes too.
As a general principle: performance gain is nice, but stability is much nicer still.
Cheers,
Franco
-
I don't think there are significant changes on 22.1.x and I'm not sure why people sometimes wonder if there are... release notes state what is being worked on and source repositories have annotated changes too.
As a general principle: performance gain is nice, but stability is much nicer still.
Cheers,
Franco
All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.
Or am I looking at the specifications of this firewall incorrectly. It mentions 14.6Gbps throughput, does that include inbound *and* outbound or should I reasonably expect full near 10Gb routing performance for traffic going both in and out of the box?
-
Maybe try these tunables and see if they are any help? The IBRS especially has helped me get more throughput on VMware.
hw.ibrs_disable:1
net.inet.rss.enabled:1
net.isr.maxthreads:-1
-
All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.
Forgive me for missing the full context here. I can't judge your setup from here, but I would assume the performance numbers given are rooted in reality for both the specifications and your measurements. The bigger question is who is going to verify why these values differ and what could be done about it.
Cheers,
Franco
-
All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.
Forgive me for missing the full context here. I can't judge your setup from here, but I would assume the performance numbers given are rooted in reality for both the specifications and your measurements. The bigger question is who is going to verify why these values differ and what could be done about it.
Cheers,
Franco
Well, Deciso are the ones that posted the numbers, are they not? If the numbers are saying that "14.6Gbps" is a total of 7Gbps coming into the box, and 7Gbps going out of the box from source to destination, then that is *incredibly* misleading especially for a 1000 euro+ piece of equipment. If I'm misinterpreting those numbers and I should, realistically, see a full near 10Gb inter-VLAN routing performance from this box (given I can get this on other routers, my switching hardware and clients aren't the problem), then there's something either wrong with my config or there's some tuning that needs to be involved.
Or perhaps is there a special version of OPNsense that these are supposed to run pre-tuned to properly achieve the advertised performance numbers?
-
All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.
Forgive me for missing the full context here. I can't judge your setup from here, but I would assume the performance numbers given are rooted in reality for both the specifications and your measurements. The bigger question is who is going to verify why these values differ and what could be done about it.
Cheers,
Franco
Well, Deciso are the ones that posted the numbers, are they not? If the numbers are saying that "14.6Gbps" is a total of 7Gbps coming into the box, and 7Gbps going out of the box from source to destination, then that is *incredibly* misleading especially for a 1000 euro+ piece of equipment. If I'm misinterpreting those numbers and I should, realistically, see a full near 10Gb inter-VLAN routing performance from this box (given I can get this on other routers, my switching hardware and clients aren't the problem), then there's something either wrong with my config or there's some tuning that needs to be involved.
Or perhaps is there a special version of OPNsense that these are supposed to run pre-tuned to properly achieve the advertised performance numbers?
Not sure what the issue is with you specifically setup, however we did notice:
- When running IPsec on the same box leads to a performance penalty.
- With the current kernel the scheduling is not optimal, resulting in a somewhat fluctuating throughput. This is resolved in the new Freebsd 13.1 kernel that has been released as beta https://forum.opnsense.org/index.php?topic=28505.0 (https://forum.opnsense.org/index.php?topic=28505.0), so feel free to test this as well.
- And obviously the online documentation has a typo where the total firewall throughput was also mentioned as port-port throughput. Since these are max 10Gbps ports, one cannot route more traffic than that. This has been corrected. Peak (see below is about 9.3 Gbps, we now list slightly below that number).
- Testing is done with spectre/meltdown mitigation disabled (default config for our firewalls), see also https://docs.opnsense.org/troubleshooting/hardening.html (https://docs.opnsense.org/troubleshooting/hardening.html)
Current version / new test
Now I just retested the performance with IPerf on the current kernel (using OPNsense® Business Edition 22.4 / should be the same as current 22.1 version) where the traffic flows through the firewall:
Test Server Port 1 --> Firewall Port 1 --> Firewall Port 2 --> Test Server Port 2
In optimal situation this results in 9.3Gbps:
# iperf3 -c 192.168.10.20 -P 8 -Z -t 10
Connecting to host 192.168.10.20, port 5201
[ 5] local 10.0.0.20 port 44956 connected to 192.168.10.20 port 5201
[ 7] local 10.0.0.20 port 44958 connected to 192.168.10.20 port 5201
[ 9] local 10.0.0.20 port 44960 connected to 192.168.10.20 port 5201
[ 11] local 10.0.0.20 port 44962 connected to 192.168.10.20 port 5201
[ 13] local 10.0.0.20 port 44964 connected to 192.168.10.20 port 5201
[ 15] local 10.0.0.20 port 44966 connected to 192.168.10.20 port 5201
[ 17] local 10.0.0.20 port 44968 connected to 192.168.10.20 port 5201
[ 19] local 10.0.0.20 port 44970 connected to 192.168.10.20 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 46.1 MBytes 387 Mbits/sec 13 249 KBytes
[ 7] 0.00-1.00 sec 313 MBytes 2.63 Gbits/sec 404 1.20 MBytes
[ 9] 0.00-1.00 sec 38.2 MBytes 320 Mbits/sec 30 226 KBytes
[ 11] 0.00-1.00 sec 43.9 MBytes 368 Mbits/sec 26 245 KBytes
[ 13] 0.00-1.00 sec 31.3 MBytes 262 Mbits/sec 14 192 KBytes
[ 15] 0.00-1.00 sec 41.0 MBytes 344 Mbits/sec 1 253 KBytes
[ 17] 0.00-1.00 sec 422 MBytes 3.54 Gbits/sec 137 1.81 MBytes
[ 19] 0.00-1.00 sec 50.2 MBytes 421 Mbits/sec 21 265 KBytes
[SUM] 0.00-1.00 sec 986 MBytes 8.27 Gbits/sec 646
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 60.5 MBytes 507 Mbits/sec 0 386 KBytes
[ 7] 1.00-2.00 sec 230 MBytes 1.93 Gbits/sec 14 975 KBytes
[ 9] 1.00-2.00 sec 54.5 MBytes 458 Mbits/sec 0 351 KBytes
[ 11] 1.00-2.00 sec 57.9 MBytes 486 Mbits/sec 0 374 KBytes
[ 13] 1.00-2.00 sec 42.7 MBytes 358 Mbits/sec 1 240 KBytes
[ 15] 1.00-2.00 sec 59.7 MBytes 501 Mbits/sec 0 379 KBytes
[ 17] 1.00-2.00 sec 403 MBytes 3.38 Gbits/sec 58 1.43 MBytes
[ 19] 1.00-2.00 sec 76.6 MBytes 643 Mbits/sec 0 427 KBytes
[SUM] 1.00-2.00 sec 985 MBytes 8.26 Gbits/sec 73
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 65.8 MBytes 552 Mbits/sec 12 387 KBytes
[ 7] 2.00-3.00 sec 171 MBytes 1.43 Gbits/sec 2 794 KBytes
[ 9] 2.00-3.00 sec 65.3 MBytes 548 Mbits/sec 6 353 KBytes
[ 11] 2.00-3.00 sec 66.3 MBytes 556 Mbits/sec 13 367 KBytes
[ 13] 2.00-3.00 sec 55.8 MBytes 468 Mbits/sec 0 366 KBytes
[ 15] 2.00-3.00 sec 82.1 MBytes 689 Mbits/sec 0 511 KBytes
[ 17] 2.00-3.00 sec 323 MBytes 2.71 Gbits/sec 103 698 KBytes
[ 19] 2.00-3.00 sec 164 MBytes 1.37 Gbits/sec 6 527 KBytes
[SUM] 2.00-3.00 sec 993 MBytes 8.33 Gbits/sec 142
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 63.1 MBytes 530 Mbits/sec 42 386 KBytes
[ 7] 3.00-4.00 sec 140 MBytes 1.17 Gbits/sec 1 663 KBytes
[ 9] 3.00-4.00 sec 65.8 MBytes 552 Mbits/sec 5 348 KBytes
[ 11] 3.00-4.00 sec 80.7 MBytes 677 Mbits/sec 0 499 KBytes
[ 13] 3.00-4.00 sec 80.5 MBytes 675 Mbits/sec 0 499 KBytes
[ 15] 3.00-4.00 sec 72.7 MBytes 610 Mbits/sec 15 345 KBytes
[ 17] 3.00-4.00 sec 276 MBytes 2.32 Gbits/sec 89 542 KBytes
[ 19] 3.00-4.00 sec 202 MBytes 1.69 Gbits/sec 111 392 KBytes
[SUM] 3.00-4.00 sec 981 MBytes 8.23 Gbits/sec 263
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 57.4 MBytes 481 Mbits/sec 60 275 KBytes
[ 7] 4.00-5.00 sec 129 MBytes 1.08 Gbits/sec 0 790 KBytes
[ 9] 4.00-5.00 sec 73.7 MBytes 619 Mbits/sec 0 475 KBytes
[ 11] 4.00-5.00 sec 77.9 MBytes 654 Mbits/sec 12 459 KBytes
[ 13] 4.00-5.00 sec 98.1 MBytes 823 Mbits/sec 11 442 KBytes
[ 15] 4.00-5.00 sec 64.3 MBytes 539 Mbits/sec 3 337 KBytes
[ 17] 4.00-5.00 sec 257 MBytes 2.15 Gbits/sec 242 291 KBytes
[ 19] 4.00-5.00 sec 217 MBytes 1.82 Gbits/sec 101 415 KBytes
[SUM] 4.00-5.00 sec 974 MBytes 8.17 Gbits/sec 429
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 50.2 MBytes 421 Mbits/sec 8 283 KBytes
[ 7] 5.00-6.00 sec 149 MBytes 1.25 Gbits/sec 0 918 KBytes
[ 9] 5.00-6.00 sec 96.8 MBytes 812 Mbits/sec 0 604 KBytes
[ 11] 5.00-6.00 sec 85.0 MBytes 713 Mbits/sec 11 421 KBytes
[ 13] 5.00-6.00 sec 70.0 MBytes 587 Mbits/sec 4 421 KBytes
[ 15] 5.00-6.00 sec 48.1 MBytes 404 Mbits/sec 25 245 KBytes
[ 17] 5.00-6.00 sec 216 MBytes 1.81 Gbits/sec 76 449 KBytes
[ 19] 5.00-6.00 sec 258 MBytes 2.16 Gbits/sec 96 421 KBytes
[SUM] 5.00-6.00 sec 973 MBytes 8.16 Gbits/sec 220
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 46.4 MBytes 389 Mbits/sec 23 307 KBytes
[ 7] 6.00-7.00 sec 145 MBytes 1.22 Gbits/sec 1 770 KBytes
[ 9] 6.00-7.00 sec 98.8 MBytes 828 Mbits/sec 22 549 KBytes
[ 11] 6.00-7.00 sec 68.8 MBytes 577 Mbits/sec 10 419 KBytes
[ 13] 6.00-7.00 sec 92.5 MBytes 776 Mbits/sec 0 556 KBytes
[ 15] 6.00-7.00 sec 44.4 MBytes 372 Mbits/sec 7 290 KBytes
[ 17] 6.00-7.00 sec 255 MBytes 2.14 Gbits/sec 65 598 KBytes
[ 19] 6.00-7.00 sec 228 MBytes 1.92 Gbits/sec 61 566 KBytes
[SUM] 6.00-7.00 sec 979 MBytes 8.22 Gbits/sec 189
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 58.6 MBytes 491 Mbits/sec 6 324 KBytes
[ 7] 7.00-8.00 sec 139 MBytes 1.16 Gbits/sec 33 650 KBytes
[ 9] 7.00-8.00 sec 87.5 MBytes 734 Mbits/sec 15 516 KBytes
[ 11] 7.00-8.00 sec 56.4 MBytes 473 Mbits/sec 9 305 KBytes
[ 13] 7.00-8.00 sec 102 MBytes 860 Mbits/sec 2 504 KBytes
[ 15] 7.00-8.00 sec 53.7 MBytes 451 Mbits/sec 15 309 KBytes
[ 17] 7.00-8.00 sec 218 MBytes 1.83 Gbits/sec 127 444 KBytes
[ 19] 7.00-8.00 sec 265 MBytes 2.22 Gbits/sec 134 503 KBytes
[SUM] 7.00-8.00 sec 981 MBytes 8.23 Gbits/sec 341
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 66.5 MBytes 557 Mbits/sec 0 449 KBytes
[ 7] 8.00-9.00 sec 108 MBytes 902 Mbits/sec 6 565 KBytes
[ 9] 8.00-9.00 sec 104 MBytes 870 Mbits/sec 17 450 KBytes
[ 11] 8.00-9.00 sec 65.7 MBytes 551 Mbits/sec 0 433 KBytes
[ 13] 8.00-9.00 sec 102 MBytes 860 Mbits/sec 6 442 KBytes
[ 15] 8.00-9.00 sec 53.9 MBytes 452 Mbits/sec 34 220 KBytes
[ 17] 8.00-9.00 sec 230 MBytes 1.93 Gbits/sec 54 441 KBytes
[ 19] 8.00-9.00 sec 251 MBytes 2.10 Gbits/sec 156 457 KBytes
[SUM] 8.00-9.00 sec 981 MBytes 8.23 Gbits/sec 273
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 91.7 MBytes 769 Mbits/sec 2 404 KBytes
[ 7] 9.00-10.00 sec 115 MBytes 965 Mbits/sec 0 692 KBytes
[ 9] 9.00-10.00 sec 93.8 MBytes 786 Mbits/sec 0 581 KBytes
[ 11] 9.00-10.00 sec 75.2 MBytes 631 Mbits/sec 11 408 KBytes
[ 13] 9.00-10.00 sec 75.0 MBytes 629 Mbits/sec 20 290 KBytes
[ 15] 9.00-10.00 sec 47.8 MBytes 401 Mbits/sec 0 341 KBytes
[ 17] 9.00-10.00 sec 236 MBytes 1.98 Gbits/sec 35 338 KBytes
[ 19] 9.00-10.00 sec 246 MBytes 2.07 Gbits/sec 62 345 KBytes
[SUM] 9.00-10.00 sec 981 MBytes 8.23 Gbits/sec 130
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 606 MBytes 509 Mbits/sec 166 sender
[ 5] 0.00-10.00 sec 604 MBytes 506 Mbits/sec receiver
[ 7] 0.00-10.00 sec 1.60 GBytes 1.37 Gbits/sec 461 sender
[ 7] 0.00-10.00 sec 1.60 GBytes 1.37 Gbits/sec receiver
[ 9] 0.00-10.00 sec 778 MBytes 653 Mbits/sec 95 sender
[ 9] 0.00-10.00 sec 776 MBytes 650 Mbits/sec receiver
[ 11] 0.00-10.00 sec 678 MBytes 569 Mbits/sec 92 sender
[ 11] 0.00-10.00 sec 675 MBytes 566 Mbits/sec receiver
[ 13] 0.00-10.00 sec 751 MBytes 630 Mbits/sec 58 sender
[ 13] 0.00-10.00 sec 747 MBytes 627 Mbits/sec receiver
[ 15] 0.00-10.00 sec 568 MBytes 476 Mbits/sec 100 sender
[ 15] 0.00-10.00 sec 565 MBytes 474 Mbits/sec receiver
[ 17] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec 986 sender
[ 17] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec receiver
[ 19] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec 748 sender
[ 19] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec receiver
[SUM] 0.00-10.00 sec 9.58 GBytes 8.23 Gbits/sec 2706 sender
[SUM] 0.00-10.00 sec 9.56 GBytes 8.21 Gbits/sec receiver
iperf Done.
root@perftest1:/opt/OPNsense_perftest# iperf3 -c 192.168.10.20 -P 8 -Z -t 10
Connecting to host 192.168.10.20, port 5201
[ 5] local 10.0.0.20 port 44974 connected to 192.168.10.20 port 5201
[ 7] local 10.0.0.20 port 44976 connected to 192.168.10.20 port 5201
[ 9] local 10.0.0.20 port 44978 connected to 192.168.10.20 port 5201
[ 11] local 10.0.0.20 port 44980 connected to 192.168.10.20 port 5201
[ 13] local 10.0.0.20 port 44982 connected to 192.168.10.20 port 5201
[ 15] local 10.0.0.20 port 44984 connected to 192.168.10.20 port 5201
[ 17] local 10.0.0.20 port 44986 connected to 192.168.10.20 port 5201
[ 19] local 10.0.0.20 port 44988 connected to 192.168.10.20 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 189 MBytes 1.59 Gbits/sec 17 449 KBytes
[ 7] 0.00-1.00 sec 91.0 MBytes 764 Mbits/sec 4 361 KBytes
[ 9] 0.00-1.00 sec 108 MBytes 905 Mbits/sec 4 392 KBytes
[ 11] 0.00-1.00 sec 164 MBytes 1.38 Gbits/sec 4 532 KBytes
[ 13] 0.00-1.00 sec 127 MBytes 1.07 Gbits/sec 5 436 KBytes
[ 15] 0.00-1.00 sec 150 MBytes 1.26 Gbits/sec 7 445 KBytes
[ 17] 0.00-1.00 sec 125 MBytes 1.05 Gbits/sec 2 382 KBytes
[ 19] 0.00-1.00 sec 178 MBytes 1.49 Gbits/sec 14 603 KBytes
[SUM] 0.00-1.00 sec 1.11 GBytes 9.50 Gbits/sec 57
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 165 MBytes 1.38 Gbits/sec 0 506 KBytes
[ 7] 1.00-2.00 sec 110 MBytes 921 Mbits/sec 0 528 KBytes
[ 9] 1.00-2.00 sec 115 MBytes 967 Mbits/sec 0 558 KBytes
[ 11] 1.00-2.00 sec 142 MBytes 1.19 Gbits/sec 0 698 KBytes
[ 13] 1.00-2.00 sec 121 MBytes 1.01 Gbits/sec 0 602 KBytes
[ 15] 1.00-2.00 sec 156 MBytes 1.31 Gbits/sec 0 567 KBytes
[ 17] 1.00-2.00 sec 154 MBytes 1.29 Gbits/sec 0 520 KBytes
[ 19] 1.00-2.00 sec 158 MBytes 1.33 Gbits/sec 0 632 KBytes
[SUM] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 160 MBytes 1.34 Gbits/sec 0 520 KBytes
[ 7] 2.00-3.00 sec 109 MBytes 918 Mbits/sec 0 662 KBytes
[ 9] 2.00-3.00 sec 113 MBytes 951 Mbits/sec 0 688 KBytes
[ 11] 2.00-3.00 sec 140 MBytes 1.17 Gbits/sec 0 827 KBytes
[ 13] 2.00-3.00 sec 120 MBytes 1.01 Gbits/sec 0 731 KBytes
[ 15] 2.00-3.00 sec 157 MBytes 1.32 Gbits/sec 0 621 KBytes
[ 17] 2.00-3.00 sec 159 MBytes 1.33 Gbits/sec 0 612 KBytes
[ 19] 2.00-3.00 sec 158 MBytes 1.33 Gbits/sec 0 674 KBytes
[SUM] 2.00-3.00 sec 1.09 GBytes 9.36 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 168 MBytes 1.41 Gbits/sec 0 544 KBytes
[ 7] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 773 KBytes
[ 9] 3.00-4.00 sec 110 MBytes 923 Mbits/sec 0 796 KBytes
[ 11] 3.00-4.00 sec 136 MBytes 1.14 Gbits/sec 0 940 KBytes
[ 13] 3.00-4.00 sec 118 MBytes 986 Mbits/sec 0 836 KBytes
[ 15] 3.00-4.00 sec 160 MBytes 1.34 Gbits/sec 0 686 KBytes
[ 17] 3.00-4.00 sec 158 MBytes 1.32 Gbits/sec 0 675 KBytes
[ 19] 3.00-4.00 sec 159 MBytes 1.33 Gbits/sec 0 707 KBytes
[SUM] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 161 MBytes 1.35 Gbits/sec 0 556 KBytes
[ 7] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 873 KBytes
[ 9] 4.00-5.00 sec 114 MBytes 954 Mbits/sec 0 891 KBytes
[ 11] 4.00-5.00 sec 135 MBytes 1.13 Gbits/sec 0 1.01 MBytes
[ 13] 4.00-5.00 sec 118 MBytes 986 Mbits/sec 12 928 KBytes
[ 15] 4.00-5.00 sec 159 MBytes 1.33 Gbits/sec 0 694 KBytes
[ 17] 4.00-5.00 sec 160 MBytes 1.34 Gbits/sec 0 685 KBytes
[ 19] 4.00-5.00 sec 159 MBytes 1.33 Gbits/sec 0 716 KBytes
[SUM] 4.00-5.00 sec 1.09 GBytes 9.37 Gbits/sec 12
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 150 MBytes 1.26 Gbits/sec 0 562 KBytes
[ 7] 5.00-6.00 sec 126 MBytes 1.06 Gbits/sec 0 968 KBytes
[ 9] 5.00-6.00 sec 126 MBytes 1.06 Gbits/sec 0 986 KBytes
[ 11] 5.00-6.00 sec 140 MBytes 1.17 Gbits/sec 0 1.10 MBytes
[ 13] 5.00-6.00 sec 129 MBytes 1.08 Gbits/sec 0 1018 KBytes
[ 15] 5.00-6.00 sec 150 MBytes 1.26 Gbits/sec 0 711 KBytes
[ 17] 5.00-6.00 sec 146 MBytes 1.23 Gbits/sec 0 700 KBytes
[ 19] 5.00-6.00 sec 150 MBytes 1.26 Gbits/sec 0 732 KBytes
[SUM] 5.00-6.00 sec 1.09 GBytes 9.37 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 567 KBytes
[ 7] 6.00-7.00 sec 139 MBytes 1.16 Gbits/sec 0 1.04 MBytes
[ 9] 6.00-7.00 sec 136 MBytes 1.14 Gbits/sec 0 1.06 MBytes
[ 11] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 1.17 MBytes
[ 13] 6.00-7.00 sec 138 MBytes 1.15 Gbits/sec 0 1.09 MBytes
[ 15] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 719 KBytes
[ 17] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 708 KBytes
[ 19] 6.00-7.00 sec 140 MBytes 1.17 Gbits/sec 0 739 KBytes
[SUM] 6.00-7.00 sec 1.09 GBytes 9.37 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 567 KBytes
[ 7] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 1.13 MBytes
[ 9] 7.00-8.00 sec 138 MBytes 1.15 Gbits/sec 0 1.14 MBytes
[ 11] 7.00-8.00 sec 142 MBytes 1.20 Gbits/sec 0 1.18 MBytes
[ 13] 7.00-8.00 sec 138 MBytes 1.15 Gbits/sec 0 1.17 MBytes
[ 15] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 724 KBytes
[ 17] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 712 KBytes
[ 19] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 742 KBytes
[SUM] 7.00-8.00 sec 1.09 GBytes 9.37 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 188 MBytes 1.58 Gbits/sec 183 563 KBytes
[ 7] 8.00-9.00 sec 132 MBytes 1.11 Gbits/sec 285 645 KBytes
[ 9] 8.00-9.00 sec 138 MBytes 1.15 Gbits/sec 270 694 KBytes
[ 11] 8.00-9.00 sec 109 MBytes 912 Mbits/sec 251 488 KBytes
[ 13] 8.00-9.00 sec 104 MBytes 870 Mbits/sec 257 477 KBytes
[ 15] 8.00-9.00 sec 150 MBytes 1.26 Gbits/sec 96 507 KBytes
[ 17] 8.00-9.00 sec 142 MBytes 1.20 Gbits/sec 254 453 KBytes
[ 19] 8.00-9.00 sec 154 MBytes 1.29 Gbits/sec 150 515 KBytes
[SUM] 8.00-9.00 sec 1.09 GBytes 9.37 Gbits/sec 1746
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 122 MBytes 1.02 Gbits/sec 0 565 KBytes
[ 7] 9.00-10.00 sec 148 MBytes 1.24 Gbits/sec 0 792 KBytes
[ 9] 9.00-10.00 sec 142 MBytes 1.20 Gbits/sec 0 823 KBytes
[ 11] 9.00-10.00 sec 98.8 MBytes 828 Mbits/sec 0 586 KBytes
[ 13] 9.00-10.00 sec 91.2 MBytes 765 Mbits/sec 0 565 KBytes
[ 15] 9.00-10.00 sec 149 MBytes 1.25 Gbits/sec 0 602 KBytes
[ 17] 9.00-10.00 sec 194 MBytes 1.63 Gbits/sec 0 636 KBytes
[ 19] 9.00-10.00 sec 120 MBytes 1.01 Gbits/sec 0 548 KBytes
[SUM] 9.00-10.00 sec 1.04 GBytes 8.93 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.55 GBytes 1.33 Gbits/sec 200 sender
[ 5] 0.00-10.00 sec 1.54 GBytes 1.33 Gbits/sec receiver
[ 7] 0.00-10.00 sec 1.19 GBytes 1.02 Gbits/sec 289 sender
[ 7] 0.00-10.00 sec 1.19 GBytes 1.02 Gbits/sec receiver
[ 9] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec 274 sender
[ 9] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec receiver
[ 11] 0.00-10.00 sec 1.32 GBytes 1.13 Gbits/sec 255 sender
[ 11] 0.00-10.00 sec 1.31 GBytes 1.13 Gbits/sec receiver
[ 13] 0.00-10.00 sec 1.17 GBytes 1.01 Gbits/sec 274 sender
[ 13] 0.00-10.00 sec 1.17 GBytes 1.01 Gbits/sec receiver
[ 15] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec 103 sender
[ 15] 0.00-10.00 sec 1.47 GBytes 1.27 Gbits/sec receiver
[ 17] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec 256 sender
[ 17] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec receiver
[ 19] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec 164 sender
[ 19] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec receiver
[SUM] 0.00-10.00 sec 10.9 GBytes 9.34 Gbits/sec 1815 sender
[SUM] 0.00-10.00 sec 10.9 GBytes 9.32 Gbits/sec receiver
iperf Done.
Retesting a couple of times does show a spread with an average of about 7-9Gbps.
With the new FreeBSD 13.1 kernel the performance averages at about 8.7Gbps (standard MTU) and fluctuates a lot less. So while a bit lower than our peak, it will likely result in higher throughput on average.
Hope this clears things up for everyone.
-
All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.
Forgive me for missing the full context here. I can't judge your setup from here, but I would assume the performance numbers given are rooted in reality for both the specifications and your measurements. The bigger question is who is going to verify why these values differ and what could be done about it.
Cheers,
Franco
Well, Deciso are the ones that posted the numbers, are they not? If the numbers are saying that "14.6Gbps" is a total of 7Gbps coming into the box, and 7Gbps going out of the box from source to destination, then that is *incredibly* misleading especially for a 1000 euro+ piece of equipment. If I'm misinterpreting those numbers and I should, realistically, see a full near 10Gb inter-VLAN routing performance from this box (given I can get this on other routers, my switching hardware and clients aren't the problem), then there's something either wrong with my config or there's some tuning that needs to be involved.
Or perhaps is there a special version of OPNsense that these are supposed to run pre-tuned to properly achieve the advertised performance numbers?
Not sure what the issue is with you specifically setup, however we did notice:
- When running IPsec on the same box leads to a performance penalty.
- With the current kernel the scheduling is not optimal, resulting in a somewhat fluctuating throughput. This is resolved in the new Freebsd 13.1 kernel that has been released as beta https://forum.opnsense.org/index.php?topic=28505.0 (https://forum.opnsense.org/index.php?topic=28505.0), so feel free to test this as well.
- And obviously the online documentation has a typo where the total firewall throughput was also mentioned as port-port throughput. Since these are max 10Gbps ports, one cannot route more traffic than that. This has been corrected. Peak (see below is about 9.3 Gbps, we now list slightly below that number).
- Testing is done with spectre/meltdown mitigation disabled (default config for our firewalls), see also https://docs.opnsense.org/troubleshooting/hardening.html (https://docs.opnsense.org/troubleshooting/hardening.html)
Current version / new test
Now I just retested the performance with IPerf on the current kernel (using OPNsense® Business Edition 22.4 / should be the same as current 22.1 version) where the traffic flows through the firewall:
Test Server Port 1 --> Firewall Port 1 --> Firewall Port 2 --> Test Server Port 2
In optimal situation this results in 9.3Gbps:
# iperf3 -c 192.168.10.20 -P 8 -Z -t 10
Connecting to host 192.168.10.20, port 5201
[ 5] local 10.0.0.20 port 44956 connected to 192.168.10.20 port 5201
[ 7] local 10.0.0.20 port 44958 connected to 192.168.10.20 port 5201
[ 9] local 10.0.0.20 port 44960 connected to 192.168.10.20 port 5201
[ 11] local 10.0.0.20 port 44962 connected to 192.168.10.20 port 5201
[ 13] local 10.0.0.20 port 44964 connected to 192.168.10.20 port 5201
[ 15] local 10.0.0.20 port 44966 connected to 192.168.10.20 port 5201
[ 17] local 10.0.0.20 port 44968 connected to 192.168.10.20 port 5201
[ 19] local 10.0.0.20 port 44970 connected to 192.168.10.20 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 46.1 MBytes 387 Mbits/sec 13 249 KBytes
[ 7] 0.00-1.00 sec 313 MBytes 2.63 Gbits/sec 404 1.20 MBytes
[ 9] 0.00-1.00 sec 38.2 MBytes 320 Mbits/sec 30 226 KBytes
[ 11] 0.00-1.00 sec 43.9 MBytes 368 Mbits/sec 26 245 KBytes
[ 13] 0.00-1.00 sec 31.3 MBytes 262 Mbits/sec 14 192 KBytes
[ 15] 0.00-1.00 sec 41.0 MBytes 344 Mbits/sec 1 253 KBytes
[ 17] 0.00-1.00 sec 422 MBytes 3.54 Gbits/sec 137 1.81 MBytes
[ 19] 0.00-1.00 sec 50.2 MBytes 421 Mbits/sec 21 265 KBytes
[SUM] 0.00-1.00 sec 986 MBytes 8.27 Gbits/sec 646
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 60.5 MBytes 507 Mbits/sec 0 386 KBytes
[ 7] 1.00-2.00 sec 230 MBytes 1.93 Gbits/sec 14 975 KBytes
[ 9] 1.00-2.00 sec 54.5 MBytes 458 Mbits/sec 0 351 KBytes
[ 11] 1.00-2.00 sec 57.9 MBytes 486 Mbits/sec 0 374 KBytes
[ 13] 1.00-2.00 sec 42.7 MBytes 358 Mbits/sec 1 240 KBytes
[ 15] 1.00-2.00 sec 59.7 MBytes 501 Mbits/sec 0 379 KBytes
[ 17] 1.00-2.00 sec 403 MBytes 3.38 Gbits/sec 58 1.43 MBytes
[ 19] 1.00-2.00 sec 76.6 MBytes 643 Mbits/sec 0 427 KBytes
[SUM] 1.00-2.00 sec 985 MBytes 8.26 Gbits/sec 73
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 65.8 MBytes 552 Mbits/sec 12 387 KBytes
[ 7] 2.00-3.00 sec 171 MBytes 1.43 Gbits/sec 2 794 KBytes
[ 9] 2.00-3.00 sec 65.3 MBytes 548 Mbits/sec 6 353 KBytes
[ 11] 2.00-3.00 sec 66.3 MBytes 556 Mbits/sec 13 367 KBytes
[ 13] 2.00-3.00 sec 55.8 MBytes 468 Mbits/sec 0 366 KBytes
[ 15] 2.00-3.00 sec 82.1 MBytes 689 Mbits/sec 0 511 KBytes
[ 17] 2.00-3.00 sec 323 MBytes 2.71 Gbits/sec 103 698 KBytes
[ 19] 2.00-3.00 sec 164 MBytes 1.37 Gbits/sec 6 527 KBytes
[SUM] 2.00-3.00 sec 993 MBytes 8.33 Gbits/sec 142
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 63.1 MBytes 530 Mbits/sec 42 386 KBytes
[ 7] 3.00-4.00 sec 140 MBytes 1.17 Gbits/sec 1 663 KBytes
[ 9] 3.00-4.00 sec 65.8 MBytes 552 Mbits/sec 5 348 KBytes
[ 11] 3.00-4.00 sec 80.7 MBytes 677 Mbits/sec 0 499 KBytes
[ 13] 3.00-4.00 sec 80.5 MBytes 675 Mbits/sec 0 499 KBytes
[ 15] 3.00-4.00 sec 72.7 MBytes 610 Mbits/sec 15 345 KBytes
[ 17] 3.00-4.00 sec 276 MBytes 2.32 Gbits/sec 89 542 KBytes
[ 19] 3.00-4.00 sec 202 MBytes 1.69 Gbits/sec 111 392 KBytes
[SUM] 3.00-4.00 sec 981 MBytes 8.23 Gbits/sec 263
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 57.4 MBytes 481 Mbits/sec 60 275 KBytes
[ 7] 4.00-5.00 sec 129 MBytes 1.08 Gbits/sec 0 790 KBytes
[ 9] 4.00-5.00 sec 73.7 MBytes 619 Mbits/sec 0 475 KBytes
[ 11] 4.00-5.00 sec 77.9 MBytes 654 Mbits/sec 12 459 KBytes
[ 13] 4.00-5.00 sec 98.1 MBytes 823 Mbits/sec 11 442 KBytes
[ 15] 4.00-5.00 sec 64.3 MBytes 539 Mbits/sec 3 337 KBytes
[ 17] 4.00-5.00 sec 257 MBytes 2.15 Gbits/sec 242 291 KBytes
[ 19] 4.00-5.00 sec 217 MBytes 1.82 Gbits/sec 101 415 KBytes
[SUM] 4.00-5.00 sec 974 MBytes 8.17 Gbits/sec 429
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 50.2 MBytes 421 Mbits/sec 8 283 KBytes
[ 7] 5.00-6.00 sec 149 MBytes 1.25 Gbits/sec 0 918 KBytes
[ 9] 5.00-6.00 sec 96.8 MBytes 812 Mbits/sec 0 604 KBytes
[ 11] 5.00-6.00 sec 85.0 MBytes 713 Mbits/sec 11 421 KBytes
[ 13] 5.00-6.00 sec 70.0 MBytes 587 Mbits/sec 4 421 KBytes
[ 15] 5.00-6.00 sec 48.1 MBytes 404 Mbits/sec 25 245 KBytes
[ 17] 5.00-6.00 sec 216 MBytes 1.81 Gbits/sec 76 449 KBytes
[ 19] 5.00-6.00 sec 258 MBytes 2.16 Gbits/sec 96 421 KBytes
[SUM] 5.00-6.00 sec 973 MBytes 8.16 Gbits/sec 220
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 46.4 MBytes 389 Mbits/sec 23 307 KBytes
[ 7] 6.00-7.00 sec 145 MBytes 1.22 Gbits/sec 1 770 KBytes
[ 9] 6.00-7.00 sec 98.8 MBytes 828 Mbits/sec 22 549 KBytes
[ 11] 6.00-7.00 sec 68.8 MBytes 577 Mbits/sec 10 419 KBytes
[ 13] 6.00-7.00 sec 92.5 MBytes 776 Mbits/sec 0 556 KBytes
[ 15] 6.00-7.00 sec 44.4 MBytes 372 Mbits/sec 7 290 KBytes
[ 17] 6.00-7.00 sec 255 MBytes 2.14 Gbits/sec 65 598 KBytes
[ 19] 6.00-7.00 sec 228 MBytes 1.92 Gbits/sec 61 566 KBytes
[SUM] 6.00-7.00 sec 979 MBytes 8.22 Gbits/sec 189
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 58.6 MBytes 491 Mbits/sec 6 324 KBytes
[ 7] 7.00-8.00 sec 139 MBytes 1.16 Gbits/sec 33 650 KBytes
[ 9] 7.00-8.00 sec 87.5 MBytes 734 Mbits/sec 15 516 KBytes
[ 11] 7.00-8.00 sec 56.4 MBytes 473 Mbits/sec 9 305 KBytes
[ 13] 7.00-8.00 sec 102 MBytes 860 Mbits/sec 2 504 KBytes
[ 15] 7.00-8.00 sec 53.7 MBytes 451 Mbits/sec 15 309 KBytes
[ 17] 7.00-8.00 sec 218 MBytes 1.83 Gbits/sec 127 444 KBytes
[ 19] 7.00-8.00 sec 265 MBytes 2.22 Gbits/sec 134 503 KBytes
[SUM] 7.00-8.00 sec 981 MBytes 8.23 Gbits/sec 341
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 66.5 MBytes 557 Mbits/sec 0 449 KBytes
[ 7] 8.00-9.00 sec 108 MBytes 902 Mbits/sec 6 565 KBytes
[ 9] 8.00-9.00 sec 104 MBytes 870 Mbits/sec 17 450 KBytes
[ 11] 8.00-9.00 sec 65.7 MBytes 551 Mbits/sec 0 433 KBytes
[ 13] 8.00-9.00 sec 102 MBytes 860 Mbits/sec 6 442 KBytes
[ 15] 8.00-9.00 sec 53.9 MBytes 452 Mbits/sec 34 220 KBytes
[ 17] 8.00-9.00 sec 230 MBytes 1.93 Gbits/sec 54 441 KBytes
[ 19] 8.00-9.00 sec 251 MBytes 2.10 Gbits/sec 156 457 KBytes
[SUM] 8.00-9.00 sec 981 MBytes 8.23 Gbits/sec 273
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 91.7 MBytes 769 Mbits/sec 2 404 KBytes
[ 7] 9.00-10.00 sec 115 MBytes 965 Mbits/sec 0 692 KBytes
[ 9] 9.00-10.00 sec 93.8 MBytes 786 Mbits/sec 0 581 KBytes
[ 11] 9.00-10.00 sec 75.2 MBytes 631 Mbits/sec 11 408 KBytes
[ 13] 9.00-10.00 sec 75.0 MBytes 629 Mbits/sec 20 290 KBytes
[ 15] 9.00-10.00 sec 47.8 MBytes 401 Mbits/sec 0 341 KBytes
[ 17] 9.00-10.00 sec 236 MBytes 1.98 Gbits/sec 35 338 KBytes
[ 19] 9.00-10.00 sec 246 MBytes 2.07 Gbits/sec 62 345 KBytes
[SUM] 9.00-10.00 sec 981 MBytes 8.23 Gbits/sec 130
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 606 MBytes 509 Mbits/sec 166 sender
[ 5] 0.00-10.00 sec 604 MBytes 506 Mbits/sec receiver
[ 7] 0.00-10.00 sec 1.60 GBytes 1.37 Gbits/sec 461 sender
[ 7] 0.00-10.00 sec 1.60 GBytes 1.37 Gbits/sec receiver
[ 9] 0.00-10.00 sec 778 MBytes 653 Mbits/sec 95 sender
[ 9] 0.00-10.00 sec 776 MBytes 650 Mbits/sec receiver
[ 11] 0.00-10.00 sec 678 MBytes 569 Mbits/sec 92 sender
[ 11] 0.00-10.00 sec 675 MBytes 566 Mbits/sec receiver
[ 13] 0.00-10.00 sec 751 MBytes 630 Mbits/sec 58 sender
[ 13] 0.00-10.00 sec 747 MBytes 627 Mbits/sec receiver
[ 15] 0.00-10.00 sec 568 MBytes 476 Mbits/sec 100 sender
[ 15] 0.00-10.00 sec 565 MBytes 474 Mbits/sec receiver
[ 17] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec 986 sender
[ 17] 0.00-10.00 sec 2.77 GBytes 2.38 Gbits/sec receiver
[ 19] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec 748 sender
[ 19] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec receiver
[SUM] 0.00-10.00 sec 9.58 GBytes 8.23 Gbits/sec 2706 sender
[SUM] 0.00-10.00 sec 9.56 GBytes 8.21 Gbits/sec receiver
iperf Done.
root@perftest1:/opt/OPNsense_perftest# iperf3 -c 192.168.10.20 -P 8 -Z -t 10
Connecting to host 192.168.10.20, port 5201
[ 5] local 10.0.0.20 port 44974 connected to 192.168.10.20 port 5201
[ 7] local 10.0.0.20 port 44976 connected to 192.168.10.20 port 5201
[ 9] local 10.0.0.20 port 44978 connected to 192.168.10.20 port 5201
[ 11] local 10.0.0.20 port 44980 connected to 192.168.10.20 port 5201
[ 13] local 10.0.0.20 port 44982 connected to 192.168.10.20 port 5201
[ 15] local 10.0.0.20 port 44984 connected to 192.168.10.20 port 5201
[ 17] local 10.0.0.20 port 44986 connected to 192.168.10.20 port 5201
[ 19] local 10.0.0.20 port 44988 connected to 192.168.10.20 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 189 MBytes 1.59 Gbits/sec 17 449 KBytes
[ 7] 0.00-1.00 sec 91.0 MBytes 764 Mbits/sec 4 361 KBytes
[ 9] 0.00-1.00 sec 108 MBytes 905 Mbits/sec 4 392 KBytes
[ 11] 0.00-1.00 sec 164 MBytes 1.38 Gbits/sec 4 532 KBytes
[ 13] 0.00-1.00 sec 127 MBytes 1.07 Gbits/sec 5 436 KBytes
[ 15] 0.00-1.00 sec 150 MBytes 1.26 Gbits/sec 7 445 KBytes
[ 17] 0.00-1.00 sec 125 MBytes 1.05 Gbits/sec 2 382 KBytes
[ 19] 0.00-1.00 sec 178 MBytes 1.49 Gbits/sec 14 603 KBytes
[SUM] 0.00-1.00 sec 1.11 GBytes 9.50 Gbits/sec 57
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 165 MBytes 1.38 Gbits/sec 0 506 KBytes
[ 7] 1.00-2.00 sec 110 MBytes 921 Mbits/sec 0 528 KBytes
[ 9] 1.00-2.00 sec 115 MBytes 967 Mbits/sec 0 558 KBytes
[ 11] 1.00-2.00 sec 142 MBytes 1.19 Gbits/sec 0 698 KBytes
[ 13] 1.00-2.00 sec 121 MBytes 1.01 Gbits/sec 0 602 KBytes
[ 15] 1.00-2.00 sec 156 MBytes 1.31 Gbits/sec 0 567 KBytes
[ 17] 1.00-2.00 sec 154 MBytes 1.29 Gbits/sec 0 520 KBytes
[ 19] 1.00-2.00 sec 158 MBytes 1.33 Gbits/sec 0 632 KBytes
[SUM] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 160 MBytes 1.34 Gbits/sec 0 520 KBytes
[ 7] 2.00-3.00 sec 109 MBytes 918 Mbits/sec 0 662 KBytes
[ 9] 2.00-3.00 sec 113 MBytes 951 Mbits/sec 0 688 KBytes
[ 11] 2.00-3.00 sec 140 MBytes 1.17 Gbits/sec 0 827 KBytes
[ 13] 2.00-3.00 sec 120 MBytes 1.01 Gbits/sec 0 731 KBytes
[ 15] 2.00-3.00 sec 157 MBytes 1.32 Gbits/sec 0 621 KBytes
[ 17] 2.00-3.00 sec 159 MBytes 1.33 Gbits/sec 0 612 KBytes
[ 19] 2.00-3.00 sec 158 MBytes 1.33 Gbits/sec 0 674 KBytes
[SUM] 2.00-3.00 sec 1.09 GBytes 9.36 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 168 MBytes 1.41 Gbits/sec 0 544 KBytes
[ 7] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 773 KBytes
[ 9] 3.00-4.00 sec 110 MBytes 923 Mbits/sec 0 796 KBytes
[ 11] 3.00-4.00 sec 136 MBytes 1.14 Gbits/sec 0 940 KBytes
[ 13] 3.00-4.00 sec 118 MBytes 986 Mbits/sec 0 836 KBytes
[ 15] 3.00-4.00 sec 160 MBytes 1.34 Gbits/sec 0 686 KBytes
[ 17] 3.00-4.00 sec 158 MBytes 1.32 Gbits/sec 0 675 KBytes
[ 19] 3.00-4.00 sec 159 MBytes 1.33 Gbits/sec 0 707 KBytes
[SUM] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 161 MBytes 1.35 Gbits/sec 0 556 KBytes
[ 7] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 873 KBytes
[ 9] 4.00-5.00 sec 114 MBytes 954 Mbits/sec 0 891 KBytes
[ 11] 4.00-5.00 sec 135 MBytes 1.13 Gbits/sec 0 1.01 MBytes
[ 13] 4.00-5.00 sec 118 MBytes 986 Mbits/sec 12 928 KBytes
[ 15] 4.00-5.00 sec 159 MBytes 1.33 Gbits/sec 0 694 KBytes
[ 17] 4.00-5.00 sec 160 MBytes 1.34 Gbits/sec 0 685 KBytes
[ 19] 4.00-5.00 sec 159 MBytes 1.33 Gbits/sec 0 716 KBytes
[SUM] 4.00-5.00 sec 1.09 GBytes 9.37 Gbits/sec 12
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 150 MBytes 1.26 Gbits/sec 0 562 KBytes
[ 7] 5.00-6.00 sec 126 MBytes 1.06 Gbits/sec 0 968 KBytes
[ 9] 5.00-6.00 sec 126 MBytes 1.06 Gbits/sec 0 986 KBytes
[ 11] 5.00-6.00 sec 140 MBytes 1.17 Gbits/sec 0 1.10 MBytes
[ 13] 5.00-6.00 sec 129 MBytes 1.08 Gbits/sec 0 1018 KBytes
[ 15] 5.00-6.00 sec 150 MBytes 1.26 Gbits/sec 0 711 KBytes
[ 17] 5.00-6.00 sec 146 MBytes 1.23 Gbits/sec 0 700 KBytes
[ 19] 5.00-6.00 sec 150 MBytes 1.26 Gbits/sec 0 732 KBytes
[SUM] 5.00-6.00 sec 1.09 GBytes 9.37 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 567 KBytes
[ 7] 6.00-7.00 sec 139 MBytes 1.16 Gbits/sec 0 1.04 MBytes
[ 9] 6.00-7.00 sec 136 MBytes 1.14 Gbits/sec 0 1.06 MBytes
[ 11] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 1.17 MBytes
[ 13] 6.00-7.00 sec 138 MBytes 1.15 Gbits/sec 0 1.09 MBytes
[ 15] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 719 KBytes
[ 17] 6.00-7.00 sec 141 MBytes 1.18 Gbits/sec 0 708 KBytes
[ 19] 6.00-7.00 sec 140 MBytes 1.17 Gbits/sec 0 739 KBytes
[SUM] 6.00-7.00 sec 1.09 GBytes 9.37 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 567 KBytes
[ 7] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 1.13 MBytes
[ 9] 7.00-8.00 sec 138 MBytes 1.15 Gbits/sec 0 1.14 MBytes
[ 11] 7.00-8.00 sec 142 MBytes 1.20 Gbits/sec 0 1.18 MBytes
[ 13] 7.00-8.00 sec 138 MBytes 1.15 Gbits/sec 0 1.17 MBytes
[ 15] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 724 KBytes
[ 17] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 712 KBytes
[ 19] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 742 KBytes
[SUM] 7.00-8.00 sec 1.09 GBytes 9.37 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 188 MBytes 1.58 Gbits/sec 183 563 KBytes
[ 7] 8.00-9.00 sec 132 MBytes 1.11 Gbits/sec 285 645 KBytes
[ 9] 8.00-9.00 sec 138 MBytes 1.15 Gbits/sec 270 694 KBytes
[ 11] 8.00-9.00 sec 109 MBytes 912 Mbits/sec 251 488 KBytes
[ 13] 8.00-9.00 sec 104 MBytes 870 Mbits/sec 257 477 KBytes
[ 15] 8.00-9.00 sec 150 MBytes 1.26 Gbits/sec 96 507 KBytes
[ 17] 8.00-9.00 sec 142 MBytes 1.20 Gbits/sec 254 453 KBytes
[ 19] 8.00-9.00 sec 154 MBytes 1.29 Gbits/sec 150 515 KBytes
[SUM] 8.00-9.00 sec 1.09 GBytes 9.37 Gbits/sec 1746
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 122 MBytes 1.02 Gbits/sec 0 565 KBytes
[ 7] 9.00-10.00 sec 148 MBytes 1.24 Gbits/sec 0 792 KBytes
[ 9] 9.00-10.00 sec 142 MBytes 1.20 Gbits/sec 0 823 KBytes
[ 11] 9.00-10.00 sec 98.8 MBytes 828 Mbits/sec 0 586 KBytes
[ 13] 9.00-10.00 sec 91.2 MBytes 765 Mbits/sec 0 565 KBytes
[ 15] 9.00-10.00 sec 149 MBytes 1.25 Gbits/sec 0 602 KBytes
[ 17] 9.00-10.00 sec 194 MBytes 1.63 Gbits/sec 0 636 KBytes
[ 19] 9.00-10.00 sec 120 MBytes 1.01 Gbits/sec 0 548 KBytes
[SUM] 9.00-10.00 sec 1.04 GBytes 8.93 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.55 GBytes 1.33 Gbits/sec 200 sender
[ 5] 0.00-10.00 sec 1.54 GBytes 1.33 Gbits/sec receiver
[ 7] 0.00-10.00 sec 1.19 GBytes 1.02 Gbits/sec 289 sender
[ 7] 0.00-10.00 sec 1.19 GBytes 1.02 Gbits/sec receiver
[ 9] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec 274 sender
[ 9] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec receiver
[ 11] 0.00-10.00 sec 1.32 GBytes 1.13 Gbits/sec 255 sender
[ 11] 0.00-10.00 sec 1.31 GBytes 1.13 Gbits/sec receiver
[ 13] 0.00-10.00 sec 1.17 GBytes 1.01 Gbits/sec 274 sender
[ 13] 0.00-10.00 sec 1.17 GBytes 1.01 Gbits/sec receiver
[ 15] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec 103 sender
[ 15] 0.00-10.00 sec 1.47 GBytes 1.27 Gbits/sec receiver
[ 17] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec 256 sender
[ 17] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec receiver
[ 19] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec 164 sender
[ 19] 0.00-10.00 sec 1.48 GBytes 1.27 Gbits/sec receiver
[SUM] 0.00-10.00 sec 10.9 GBytes 9.34 Gbits/sec 1815 sender
[SUM] 0.00-10.00 sec 10.9 GBytes 9.32 Gbits/sec receiver
iperf Done.
Retesting a couple of times does show a spread with an average of about 7-9Gbps.
With the new FreeBSD 13.1 kernel the performance averages at about 8.7Gbps (standard MTU) and fluctuates a lot less. So while a bit lower than our peak, it will likely result in higher throughput on average.
Hope this clears things up for everyone.
And is this with a DEC3840? If you're seeing 7-9Gbps average, where does the 14.6Gbps firewall throughput number come from?
-
And is this with a DEC3840? If you're seeing 7-9Gbps average, where does the 14.6Gbps firewall throughput number come from?
Yes, the results are from the DEC3840.
Total firewall throughput is the maximum the firewall can handle and is calculated by saturating the firewall with small packets (full system utilisation) multiplied with a standard package size of 1500 bytes. So in this case we measured a peak performance of 1200KPPS multiplied with 1500 bytes, leaves us 14.4Gbps.
By the way this is with the firewall enabled, routing only performance is higher (disable the firewall).
-
And is this with a DEC3840? If you're seeing 7-9Gbps average, where does the 14.6Gbps firewall throughput number come from?
Yes, the results are from the DEC3840.
Total firewall throughput is the maximum the firewall can handle and is calculated by saturating the firewall with small packets (full system utilisation) multiplied with a standard package size of 1500 bytes. So in this case we measured a peak performance of 1200KPPS multiplied with 1500 bytes, leaves us 14.4Gbps.
By the way this is with the firewall enabled, routing only performance is higher (disable the firewall).
So...would a 21.x config restored to 22.x cause slow performance? Is there anything to try for troubleshooting purposes when it comes to solving these issues? I can confirm layer 2 is not an issue, seeing full 10Gb across Layer 2, and like I mentioned, from the firewall itself out over Layer 3 to a client device is seeing full 10Gb.