Messages

Hi NW4FUN,

I would not recommend changing the fan yourself.
Perhaps it's best to contact us at support@opnsense.com.

Also, I don't know when you bought this device, but the current models have a low noise fan.
The other alternative would be our desktop range (DEC840), essentially the same product but in a fan-less desktop enclosure.


Cheers,

Jos

And is this with a DEC3840? If you're seeing 7-9Gbps average, where does the 14.6Gbps firewall throughput number come from?

Yes, the results are from the DEC3840.
Total firewall throughput is the maximum the firewall can handle and is calculated by saturating the firewall with small packets (full system utilisation) multiplied with a standard package size of 1500 bytes. So in this case we measured a peak performance of 1200KPPS multiplied with 1500 bytes, leaves us 14.4Gbps.

By the way this is with the firewall enabled, routing only performance is higher (disable the firewall).

[Current version / new test]

All fine and well, but we're still (I am at least, I presume others are too) experiencing some performance issues.

Forgive me for missing the full context here. I can't judge your setup from here, but I would assume the performance numbers given are rooted in reality for both the specifications and your measurements. The bigger question is who is going to verify why these values differ and what could be done about it.


Cheers,
Franco

Well, Deciso are the ones that posted the numbers, are they not? If the numbers are saying that "14.6Gbps" is a total of 7Gbps coming into the box, and 7Gbps going out of the box from source to destination, then that is *incredibly* misleading especially for a 1000 euro+ piece of equipment. If I'm misinterpreting those numbers and I should, realistically, see a full near 10Gb inter-VLAN routing performance from this box (given I can get this on other routers, my switching hardware and clients aren't the problem), then there's something either wrong with my config or there's some tuning that needs to be involved.

Or perhaps is there a special version of OPNsense that these are supposed to run pre-tuned to properly achieve the advertised performance numbers?

Not sure what the issue is with you specifically setup, however we did notice:

• When running IPsec on the same box leads to a performance penalty.
  
• With the current kernel the scheduling is not optimal, resulting in a somewhat fluctuating throughput. This is resolved in the new Freebsd 13.1 kernel that has been released as beta https://forum.opnsense.org/index.php?topic=28505.0, so feel free to test this as well.
• And obviously the online documentation has a typo where the total firewall throughput was also mentioned as port-port throughput. Since these are max 10Gbps ports, one cannot route more traffic than that. This has been corrected. Peak (see below is about 9.3 Gbps, we now list slightly below that number).
• Testing is done with spectre/meltdown mitigation disabled (default config for our firewalls), see also https://docs.opnsense.org/troubleshooting/hardening.html

Current version / new test
Now I just retested the performance with IPerf on the current kernel (using OPNsense® Business Edition 22.4 / should be the same as current 22.1 version) where the traffic flows through the firewall:

Test Server Port 1 --> Firewall Port 1 --> Firewall Port 2 --> Test Server Port 2

In optimal situation this results in 9.3Gbps:

# iperf3 -c 192.168.10.20 -P 8 -Z -t 10
Connecting to host 192.168.10.20, port 5201
[  5] local 10.0.0.20 port 44956 connected to 192.168.10.20 port 5201
[  7] local 10.0.0.20 port 44958 connected to 192.168.10.20 port 5201
[  9] local 10.0.0.20 port 44960 connected to 192.168.10.20 port 5201
[ 11] local 10.0.0.20 port 44962 connected to 192.168.10.20 port 5201
[ 13] local 10.0.0.20 port 44964 connected to 192.168.10.20 port 5201
[ 15] local 10.0.0.20 port 44966 connected to 192.168.10.20 port 5201
[ 17] local 10.0.0.20 port 44968 connected to 192.168.10.20 port 5201
[ 19] local 10.0.0.20 port 44970 connected to 192.168.10.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  46.1 MBytes   387 Mbits/sec   13    249 KBytes       
[  7]   0.00-1.00   sec   313 MBytes  2.63 Gbits/sec  404   1.20 MBytes       
[  9]   0.00-1.00   sec  38.2 MBytes   320 Mbits/sec   30    226 KBytes       
[ 11]   0.00-1.00   sec  43.9 MBytes   368 Mbits/sec   26    245 KBytes       
[ 13]   0.00-1.00   sec  31.3 MBytes   262 Mbits/sec   14    192 KBytes       
[ 15]   0.00-1.00   sec  41.0 MBytes   344 Mbits/sec    1    253 KBytes       
[ 17]   0.00-1.00   sec   422 MBytes  3.54 Gbits/sec  137   1.81 MBytes       
[ 19]   0.00-1.00   sec  50.2 MBytes   421 Mbits/sec   21    265 KBytes       
[SUM]   0.00-1.00   sec   986 MBytes  8.27 Gbits/sec  646             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  60.5 MBytes   507 Mbits/sec    0    386 KBytes       
[  7]   1.00-2.00   sec   230 MBytes  1.93 Gbits/sec   14    975 KBytes       
[  9]   1.00-2.00   sec  54.5 MBytes   458 Mbits/sec    0    351 KBytes       
[ 11]   1.00-2.00   sec  57.9 MBytes   486 Mbits/sec    0    374 KBytes       
[ 13]   1.00-2.00   sec  42.7 MBytes   358 Mbits/sec    1    240 KBytes       
[ 15]   1.00-2.00   sec  59.7 MBytes   501 Mbits/sec    0    379 KBytes       
[ 17]   1.00-2.00   sec   403 MBytes  3.38 Gbits/sec   58   1.43 MBytes       
[ 19]   1.00-2.00   sec  76.6 MBytes   643 Mbits/sec    0    427 KBytes       
[SUM]   1.00-2.00   sec   985 MBytes  8.26 Gbits/sec   73             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  65.8 MBytes   552 Mbits/sec   12    387 KBytes       
[  7]   2.00-3.00   sec   171 MBytes  1.43 Gbits/sec    2    794 KBytes       
[  9]   2.00-3.00   sec  65.3 MBytes   548 Mbits/sec    6    353 KBytes       
[ 11]   2.00-3.00   sec  66.3 MBytes   556 Mbits/sec   13    367 KBytes       
[ 13]   2.00-3.00   sec  55.8 MBytes   468 Mbits/sec    0    366 KBytes       
[ 15]   2.00-3.00   sec  82.1 MBytes   689 Mbits/sec    0    511 KBytes       
[ 17]   2.00-3.00   sec   323 MBytes  2.71 Gbits/sec  103    698 KBytes       
[ 19]   2.00-3.00   sec   164 MBytes  1.37 Gbits/sec    6    527 KBytes       
[SUM]   2.00-3.00   sec   993 MBytes  8.33 Gbits/sec  142             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  63.1 MBytes   530 Mbits/sec   42    386 KBytes       
[  7]   3.00-4.00   sec   140 MBytes  1.17 Gbits/sec    1    663 KBytes       
[  9]   3.00-4.00   sec  65.8 MBytes   552 Mbits/sec    5    348 KBytes       
[ 11]   3.00-4.00   sec  80.7 MBytes   677 Mbits/sec    0    499 KBytes       
[ 13]   3.00-4.00   sec  80.5 MBytes   675 Mbits/sec    0    499 KBytes       
[ 15]   3.00-4.00   sec  72.7 MBytes   610 Mbits/sec   15    345 KBytes       
[ 17]   3.00-4.00   sec   276 MBytes  2.32 Gbits/sec   89    542 KBytes       
[ 19]   3.00-4.00   sec   202 MBytes  1.69 Gbits/sec  111    392 KBytes       
[SUM]   3.00-4.00   sec   981 MBytes  8.23 Gbits/sec  263             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  57.4 MBytes   481 Mbits/sec   60    275 KBytes       
[  7]   4.00-5.00   sec   129 MBytes  1.08 Gbits/sec    0    790 KBytes       
[  9]   4.00-5.00   sec  73.7 MBytes   619 Mbits/sec    0    475 KBytes       
[ 11]   4.00-5.00   sec  77.9 MBytes   654 Mbits/sec   12    459 KBytes       
[ 13]   4.00-5.00   sec  98.1 MBytes   823 Mbits/sec   11    442 KBytes       
[ 15]   4.00-5.00   sec  64.3 MBytes   539 Mbits/sec    3    337 KBytes       
[ 17]   4.00-5.00   sec   257 MBytes  2.15 Gbits/sec  242    291 KBytes       
[ 19]   4.00-5.00   sec   217 MBytes  1.82 Gbits/sec  101    415 KBytes       
[SUM]   4.00-5.00   sec   974 MBytes  8.17 Gbits/sec  429             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  50.2 MBytes   421 Mbits/sec    8    283 KBytes       
[  7]   5.00-6.00   sec   149 MBytes  1.25 Gbits/sec    0    918 KBytes       
[  9]   5.00-6.00   sec  96.8 MBytes   812 Mbits/sec    0    604 KBytes       
[ 11]   5.00-6.00   sec  85.0 MBytes   713 Mbits/sec   11    421 KBytes       
[ 13]   5.00-6.00   sec  70.0 MBytes   587 Mbits/sec    4    421 KBytes       
[ 15]   5.00-6.00   sec  48.1 MBytes   404 Mbits/sec   25    245 KBytes       
[ 17]   5.00-6.00   sec   216 MBytes  1.81 Gbits/sec   76    449 KBytes       
[ 19]   5.00-6.00   sec   258 MBytes  2.16 Gbits/sec   96    421 KBytes       
[SUM]   5.00-6.00   sec   973 MBytes  8.16 Gbits/sec  220             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  46.4 MBytes   389 Mbits/sec   23    307 KBytes       
[  7]   6.00-7.00   sec   145 MBytes  1.22 Gbits/sec    1    770 KBytes       
[  9]   6.00-7.00   sec  98.8 MBytes   828 Mbits/sec   22    549 KBytes       
[ 11]   6.00-7.00   sec  68.8 MBytes   577 Mbits/sec   10    419 KBytes       
[ 13]   6.00-7.00   sec  92.5 MBytes   776 Mbits/sec    0    556 KBytes       
[ 15]   6.00-7.00   sec  44.4 MBytes   372 Mbits/sec    7    290 KBytes       
[ 17]   6.00-7.00   sec   255 MBytes  2.14 Gbits/sec   65    598 KBytes       
[ 19]   6.00-7.00   sec   228 MBytes  1.92 Gbits/sec   61    566 KBytes       
[SUM]   6.00-7.00   sec   979 MBytes  8.22 Gbits/sec  189             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  58.6 MBytes   491 Mbits/sec    6    324 KBytes       
[  7]   7.00-8.00   sec   139 MBytes  1.16 Gbits/sec   33    650 KBytes       
[  9]   7.00-8.00   sec  87.5 MBytes   734 Mbits/sec   15    516 KBytes       
[ 11]   7.00-8.00   sec  56.4 MBytes   473 Mbits/sec    9    305 KBytes       
[ 13]   7.00-8.00   sec   102 MBytes   860 Mbits/sec    2    504 KBytes       
[ 15]   7.00-8.00   sec  53.7 MBytes   451 Mbits/sec   15    309 KBytes       
[ 17]   7.00-8.00   sec   218 MBytes  1.83 Gbits/sec  127    444 KBytes       
[ 19]   7.00-8.00   sec   265 MBytes  2.22 Gbits/sec  134    503 KBytes       
[SUM]   7.00-8.00   sec   981 MBytes  8.23 Gbits/sec  341             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  66.5 MBytes   557 Mbits/sec    0    449 KBytes       
[  7]   8.00-9.00   sec   108 MBytes   902 Mbits/sec    6    565 KBytes       
[  9]   8.00-9.00   sec   104 MBytes   870 Mbits/sec   17    450 KBytes       
[ 11]   8.00-9.00   sec  65.7 MBytes   551 Mbits/sec    0    433 KBytes       
[ 13]   8.00-9.00   sec   102 MBytes   860 Mbits/sec    6    442 KBytes       
[ 15]   8.00-9.00   sec  53.9 MBytes   452 Mbits/sec   34    220 KBytes       
[ 17]   8.00-9.00   sec   230 MBytes  1.93 Gbits/sec   54    441 KBytes       
[ 19]   8.00-9.00   sec   251 MBytes  2.10 Gbits/sec  156    457 KBytes       
[SUM]   8.00-9.00   sec   981 MBytes  8.23 Gbits/sec  273             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  91.7 MBytes   769 Mbits/sec    2    404 KBytes       
[  7]   9.00-10.00  sec   115 MBytes   965 Mbits/sec    0    692 KBytes       
[  9]   9.00-10.00  sec  93.8 MBytes   786 Mbits/sec    0    581 KBytes       
[ 11]   9.00-10.00  sec  75.2 MBytes   631 Mbits/sec   11    408 KBytes       
[ 13]   9.00-10.00  sec  75.0 MBytes   629 Mbits/sec   20    290 KBytes       
[ 15]   9.00-10.00  sec  47.8 MBytes   401 Mbits/sec    0    341 KBytes       
[ 17]   9.00-10.00  sec   236 MBytes  1.98 Gbits/sec   35    338 KBytes       
[ 19]   9.00-10.00  sec   246 MBytes  2.07 Gbits/sec   62    345 KBytes       
[SUM]   9.00-10.00  sec   981 MBytes  8.23 Gbits/sec  130             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   606 MBytes   509 Mbits/sec  166             sender
[  5]   0.00-10.00  sec   604 MBytes   506 Mbits/sec                  receiver
[  7]   0.00-10.00  sec  1.60 GBytes  1.37 Gbits/sec  461             sender
[  7]   0.00-10.00  sec  1.60 GBytes  1.37 Gbits/sec                  receiver
[  9]   0.00-10.00  sec   778 MBytes   653 Mbits/sec   95             sender
[  9]   0.00-10.00  sec   776 MBytes   650 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   678 MBytes   569 Mbits/sec   92             sender
[ 11]   0.00-10.00  sec   675 MBytes   566 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec   751 MBytes   630 Mbits/sec   58             sender
[ 13]   0.00-10.00  sec   747 MBytes   627 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec   568 MBytes   476 Mbits/sec  100             sender
[ 15]   0.00-10.00  sec   565 MBytes   474 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec  986             sender
[ 17]   0.00-10.00  sec  2.77 GBytes  2.38 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  1.91 GBytes  1.64 Gbits/sec  748             sender
[ 19]   0.00-10.00  sec  1.91 GBytes  1.64 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  9.58 GBytes  8.23 Gbits/sec  2706             sender
[SUM]   0.00-10.00  sec  9.56 GBytes  8.21 Gbits/sec                  receiver

iperf Done.
root@perftest1:/opt/OPNsense_perftest# iperf3 -c 192.168.10.20 -P 8 -Z -t 10
Connecting to host 192.168.10.20, port 5201
[  5] local 10.0.0.20 port 44974 connected to 192.168.10.20 port 5201
[  7] local 10.0.0.20 port 44976 connected to 192.168.10.20 port 5201
[  9] local 10.0.0.20 port 44978 connected to 192.168.10.20 port 5201
[ 11] local 10.0.0.20 port 44980 connected to 192.168.10.20 port 5201
[ 13] local 10.0.0.20 port 44982 connected to 192.168.10.20 port 5201
[ 15] local 10.0.0.20 port 44984 connected to 192.168.10.20 port 5201
[ 17] local 10.0.0.20 port 44986 connected to 192.168.10.20 port 5201
[ 19] local 10.0.0.20 port 44988 connected to 192.168.10.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   189 MBytes  1.59 Gbits/sec   17    449 KBytes       
[  7]   0.00-1.00   sec  91.0 MBytes   764 Mbits/sec    4    361 KBytes       
[  9]   0.00-1.00   sec   108 MBytes   905 Mbits/sec    4    392 KBytes       
[ 11]   0.00-1.00   sec   164 MBytes  1.38 Gbits/sec    4    532 KBytes       
[ 13]   0.00-1.00   sec   127 MBytes  1.07 Gbits/sec    5    436 KBytes       
[ 15]   0.00-1.00   sec   150 MBytes  1.26 Gbits/sec    7    445 KBytes       
[ 17]   0.00-1.00   sec   125 MBytes  1.05 Gbits/sec    2    382 KBytes       
[ 19]   0.00-1.00   sec   178 MBytes  1.49 Gbits/sec   14    603 KBytes       
[SUM]   0.00-1.00   sec  1.11 GBytes  9.50 Gbits/sec   57             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   165 MBytes  1.38 Gbits/sec    0    506 KBytes       
[  7]   1.00-2.00   sec   110 MBytes   921 Mbits/sec    0    528 KBytes       
[  9]   1.00-2.00   sec   115 MBytes   967 Mbits/sec    0    558 KBytes       
[ 11]   1.00-2.00   sec   142 MBytes  1.19 Gbits/sec    0    698 KBytes       
[ 13]   1.00-2.00   sec   121 MBytes  1.01 Gbits/sec    0    602 KBytes       
[ 15]   1.00-2.00   sec   156 MBytes  1.31 Gbits/sec    0    567 KBytes       
[ 17]   1.00-2.00   sec   154 MBytes  1.29 Gbits/sec    0    520 KBytes       
[ 19]   1.00-2.00   sec   158 MBytes  1.33 Gbits/sec    0    632 KBytes       
[SUM]   1.00-2.00   sec  1.09 GBytes  9.40 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   160 MBytes  1.34 Gbits/sec    0    520 KBytes       
[  7]   2.00-3.00   sec   109 MBytes   918 Mbits/sec    0    662 KBytes       
[  9]   2.00-3.00   sec   113 MBytes   951 Mbits/sec    0    688 KBytes       
[ 11]   2.00-3.00   sec   140 MBytes  1.17 Gbits/sec    0    827 KBytes       
[ 13]   2.00-3.00   sec   120 MBytes  1.01 Gbits/sec    0    731 KBytes       
[ 15]   2.00-3.00   sec   157 MBytes  1.32 Gbits/sec    0    621 KBytes       
[ 17]   2.00-3.00   sec   159 MBytes  1.33 Gbits/sec    0    612 KBytes       
[ 19]   2.00-3.00   sec   158 MBytes  1.33 Gbits/sec    0    674 KBytes       
[SUM]   2.00-3.00   sec  1.09 GBytes  9.36 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   168 MBytes  1.41 Gbits/sec    0    544 KBytes       
[  7]   3.00-4.00   sec   112 MBytes   944 Mbits/sec    0    773 KBytes       
[  9]   3.00-4.00   sec   110 MBytes   923 Mbits/sec    0    796 KBytes       
[ 11]   3.00-4.00   sec   136 MBytes  1.14 Gbits/sec    0    940 KBytes       
[ 13]   3.00-4.00   sec   118 MBytes   986 Mbits/sec    0    836 KBytes       
[ 15]   3.00-4.00   sec   160 MBytes  1.34 Gbits/sec    0    686 KBytes       
[ 17]   3.00-4.00   sec   158 MBytes  1.32 Gbits/sec    0    675 KBytes       
[ 19]   3.00-4.00   sec   159 MBytes  1.33 Gbits/sec    0    707 KBytes       
[SUM]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   161 MBytes  1.35 Gbits/sec    0    556 KBytes       
[  7]   4.00-5.00   sec   112 MBytes   944 Mbits/sec    0    873 KBytes       
[  9]   4.00-5.00   sec   114 MBytes   954 Mbits/sec    0    891 KBytes       
[ 11]   4.00-5.00   sec   135 MBytes  1.13 Gbits/sec    0   1.01 MBytes       
[ 13]   4.00-5.00   sec   118 MBytes   986 Mbits/sec   12    928 KBytes       
[ 15]   4.00-5.00   sec   159 MBytes  1.33 Gbits/sec    0    694 KBytes       
[ 17]   4.00-5.00   sec   160 MBytes  1.34 Gbits/sec    0    685 KBytes       
[ 19]   4.00-5.00   sec   159 MBytes  1.33 Gbits/sec    0    716 KBytes       
[SUM]   4.00-5.00   sec  1.09 GBytes  9.37 Gbits/sec   12             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   150 MBytes  1.26 Gbits/sec    0    562 KBytes       
[  7]   5.00-6.00   sec   126 MBytes  1.06 Gbits/sec    0    968 KBytes       
[  9]   5.00-6.00   sec   126 MBytes  1.06 Gbits/sec    0    986 KBytes       
[ 11]   5.00-6.00   sec   140 MBytes  1.17 Gbits/sec    0   1.10 MBytes       
[ 13]   5.00-6.00   sec   129 MBytes  1.08 Gbits/sec    0   1018 KBytes       
[ 15]   5.00-6.00   sec   150 MBytes  1.26 Gbits/sec    0    711 KBytes       
[ 17]   5.00-6.00   sec   146 MBytes  1.23 Gbits/sec    0    700 KBytes       
[ 19]   5.00-6.00   sec   150 MBytes  1.26 Gbits/sec    0    732 KBytes       
[SUM]   5.00-6.00   sec  1.09 GBytes  9.37 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   141 MBytes  1.18 Gbits/sec    0    567 KBytes       
[  7]   6.00-7.00   sec   139 MBytes  1.16 Gbits/sec    0   1.04 MBytes       
[  9]   6.00-7.00   sec   136 MBytes  1.14 Gbits/sec    0   1.06 MBytes       
[ 11]   6.00-7.00   sec   141 MBytes  1.18 Gbits/sec    0   1.17 MBytes       
[ 13]   6.00-7.00   sec   138 MBytes  1.15 Gbits/sec    0   1.09 MBytes       
[ 15]   6.00-7.00   sec   141 MBytes  1.18 Gbits/sec    0    719 KBytes       
[ 17]   6.00-7.00   sec   141 MBytes  1.18 Gbits/sec    0    708 KBytes       
[ 19]   6.00-7.00   sec   140 MBytes  1.17 Gbits/sec    0    739 KBytes       
[SUM]   6.00-7.00   sec  1.09 GBytes  9.37 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   140 MBytes  1.17 Gbits/sec    0    567 KBytes       
[  7]   7.00-8.00   sec   140 MBytes  1.17 Gbits/sec    0   1.13 MBytes       
[  9]   7.00-8.00   sec   138 MBytes  1.15 Gbits/sec    0   1.14 MBytes       
[ 11]   7.00-8.00   sec   142 MBytes  1.20 Gbits/sec    0   1.18 MBytes       
[ 13]   7.00-8.00   sec   138 MBytes  1.15 Gbits/sec    0   1.17 MBytes       
[ 15]   7.00-8.00   sec   140 MBytes  1.17 Gbits/sec    0    724 KBytes       
[ 17]   7.00-8.00   sec   140 MBytes  1.17 Gbits/sec    0    712 KBytes       
[ 19]   7.00-8.00   sec   140 MBytes  1.17 Gbits/sec    0    742 KBytes       
[SUM]   7.00-8.00   sec  1.09 GBytes  9.37 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   188 MBytes  1.58 Gbits/sec  183    563 KBytes       
[  7]   8.00-9.00   sec   132 MBytes  1.11 Gbits/sec  285    645 KBytes       
[  9]   8.00-9.00   sec   138 MBytes  1.15 Gbits/sec  270    694 KBytes       
[ 11]   8.00-9.00   sec   109 MBytes   912 Mbits/sec  251    488 KBytes       
[ 13]   8.00-9.00   sec   104 MBytes   870 Mbits/sec  257    477 KBytes       
[ 15]   8.00-9.00   sec   150 MBytes  1.26 Gbits/sec   96    507 KBytes       
[ 17]   8.00-9.00   sec   142 MBytes  1.20 Gbits/sec  254    453 KBytes       
[ 19]   8.00-9.00   sec   154 MBytes  1.29 Gbits/sec  150    515 KBytes       
[SUM]   8.00-9.00   sec  1.09 GBytes  9.37 Gbits/sec  1746             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   122 MBytes  1.02 Gbits/sec    0    565 KBytes       
[  7]   9.00-10.00  sec   148 MBytes  1.24 Gbits/sec    0    792 KBytes       
[  9]   9.00-10.00  sec   142 MBytes  1.20 Gbits/sec    0    823 KBytes       
[ 11]   9.00-10.00  sec  98.8 MBytes   828 Mbits/sec    0    586 KBytes       
[ 13]   9.00-10.00  sec  91.2 MBytes   765 Mbits/sec    0    565 KBytes       
[ 15]   9.00-10.00  sec   149 MBytes  1.25 Gbits/sec    0    602 KBytes       
[ 17]   9.00-10.00  sec   194 MBytes  1.63 Gbits/sec    0    636 KBytes       
[ 19]   9.00-10.00  sec   120 MBytes  1.01 Gbits/sec    0    548 KBytes       
[SUM]   9.00-10.00  sec  1.04 GBytes  8.93 Gbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.55 GBytes  1.33 Gbits/sec  200             sender
[  5]   0.00-10.00  sec  1.54 GBytes  1.33 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  1.19 GBytes  1.02 Gbits/sec  289             sender
[  7]   0.00-10.00  sec  1.19 GBytes  1.02 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  1.21 GBytes  1.04 Gbits/sec  274             sender
[  9]   0.00-10.00  sec  1.21 GBytes  1.04 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  1.32 GBytes  1.13 Gbits/sec  255             sender
[ 11]   0.00-10.00  sec  1.31 GBytes  1.13 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  1.17 GBytes  1.01 Gbits/sec  274             sender
[ 13]   0.00-10.00  sec  1.17 GBytes  1.01 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  1.48 GBytes  1.27 Gbits/sec  103             sender
[ 15]   0.00-10.00  sec  1.47 GBytes  1.27 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  1.48 GBytes  1.27 Gbits/sec  256             sender
[ 17]   0.00-10.00  sec  1.48 GBytes  1.27 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  1.48 GBytes  1.27 Gbits/sec  164             sender
[ 19]   0.00-10.00  sec  1.48 GBytes  1.27 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  10.9 GBytes  9.34 Gbits/sec  1815             sender
[SUM]   0.00-10.00  sec  10.9 GBytes  9.32 Gbits/sec                  receiver

iperf Done.

Retesting a couple of times does show a spread with an average of about 7-9Gbps.

With the new FreeBSD 13.1 kernel the performance averages at about 8.7Gbps (standard MTU) and fluctuates a lot less. So while a bit lower than our peak, it will likely result in higher throughput on average.

Hope this clears things up for everyone.

We have tested a couple of miniPCIe LTE cards some time ago, you can find them listed here: https://docs.opnsense.org/manual/mobile_wan.html

In general if FreeBSD/HardenedBSD supports a card then it will most likely work under OPNsense too.

@newsense, as said by others before an open source project is a community effort. This means that if you want to catch issues before a release it relies fully on your input and that of others.

The core team cannot test on any exotic piece of hardware, you will have to do that work yourself.

There are a lot of people participating in testing and running the development version long before it's release, they are reporting issues and help getting them fixed.

We are grateful for everyone putting in the work required to make this project as great as it is.

For those looking for professional support, just buy support from us.. we'll help you regardless of your hardware choice, but keep in mind that bug fixing is time consuming work.

To determine the cause of the performance drop (probably a large ruleset containing patterns) I would suggest disabling all that contain patterns and then re-enable one by one.

Also ssl fingerprint rules are very consuming, this will likely be fixed with  Suricata 4.1 in the upcoming OPNsense 19.1 release.

So experimenting with enabling/disabling rulesets may be the best way to figure this out.

In general you need a performant multi core CPU for high throughput when a lot of pattern matching and/or ssl fingerprint rules are enabled.

Yes that are the ET Open rules, these are free/community rules.
The ET PRO are the commercial rules.

I am not sure why hyperscan seems to crash with certain rules, should investigate that.. but lacking time.
With the ETOpen/Pro rules it seems to function fine...

Issues could be related to the type and amount of pcre rules, memory exhaustion could be a cause..
As for performance difference, hyperscan only makes sense on pattern matching rules.

Aliases can also get copy/pasted from a comma separated list.

The services of abuse.ch are down, see https://twitter.com/abuse_ch

SSLBL / SinkDB Update: I'm sorry to tell you guys that the said services (incl. the Feodo Tracker blacklist) will be down until at least Monday, Nov 19th 2018. The reason for the outage is a failed Debian dist upgrade on the backend server. Sorry for any inconvinience.

For patern matching hyperscan is faster, but all depends on the rules used.
More information can be found here (mind that the current version of Suricata is more performant that the tested version in this doc): https://www.intel.com/content/dam/www/public/us/en/documents/solution-briefs/hyperscan-scalability-solution-brief.pdf

Could be an issue with a ruleset.
Some rulesets do not "like" Hyperscan, so you may want to try Aho-Corasick as a pattered matcher.

[Developer Open Source Security]

Developer Open Source Security

Do you want to be part of one of the fastest growing open source security projects and influence the development of a product that secures thousands of networks worldwide?
Become a developer at Deciso and use your knowledge and experience to make the world an even safer place by working on our product.

OPNsense is a security platform aimed at offering commonly used network protocols and security measures to end users or automated processes. To make this possible, all kinds of challenging techniques are used in the field of both backend and frontend development.

Together with developers from our organization and the community around the project, you will work on improving and expanding OPNsense.

What do we expect from an OPNsense developer?
•   A completed hbo study;
•   Affinity with networks and security;
•   Analytical thinking;
•   Good control of the Dutch and English languages.
•   Knowledge and experience in Python, PHP and JavaScript;

What we can offer you
Besides a challenging job at a competitive salary and fringe benefits, we offer a nice team with many opportunities to work on your personal development.
Our place of employment is Middelharnis, working from home is possible in consultation.

About Deciso
Deciso is a technology company that manufactures and sells network appliances and network security software.
Deciso was founded in 2000 and has a strong focus on open source technology.

Our security solutions provide protection against network threats in the broadest sense of the word and form the first line of defense against malware, trojans, DDOS attacks and various forms of fraud.

Part of the strategy, and the breeding ground for many technical innovations, is the fast-growing OPNsense open source firewall project. From this project, techniques relating to high-availability, VPN (secure access for branch offices and itinerant employees), intrusion detection/prevention have been (re)developed and made applicable to a broad public. The security software is made available, among other things, on a hardware platform developed and produced in the Netherlands.

Want to know more?
Then contact Robert van Papeveld, +31 (0)187-744 020.

Application
We would like to receive your application at career@deciso.com

[Developer Open Source Security]

Developer Open Source Security

Wil je onderdeel zijn van één van de snelst groeiende open source security projecten en invloed hebben op de ontwikkeling van een product dat duizenden netwerken wereldwijd beveiligd?
Wordt dan ontwikkelaar bij Deciso en gebruik jouw kennis en ervaring om de wereld nog veiliger te maken door mee te werken aan ons product.

OPNsense is een beveiligingsplatform gericht op het aanbieden van veel gebruikte netwerkprotocollen en beveiligingsmaatregelen aan eindgebruikers of geautomatiseerde processen. Om dit mogelijk te maken worden allerlei uitdagende technieken gebruikt op het gebied van zowel backend als frontend ontwikkeling.

Samen met ontwikkelaars vanuit onze organisatie en de community om het project heen werk je aan het verbeteren en uitbreiden van OPNsense.

Wat vragen wij van een OPNsense developer
•   Een afgeronde Hbo-opleiding;
•   Affiniteit met netwerken en beveiliging
•   Analytisch denkvermogen
•   Goede beheersing van de Nederlandse en Engelse taal
•   Kennis en ervaring in Python, PHP en JavaScript;

Wat kunnen wij je bieden
Naast een uitdagende functie tegen een marktconform salaris en secundaire arbeidsvoorwaarden bieden wij een leuk team met veel mogelijkheden om te werken aan je persoonlijke ontwikkeling.
Onze standplaats is Middelharnis, werken vanuit huis behoort in overleg tot de mogelijkheden.

Over Deciso
Deciso is een technologiebedrijf dat netwerk appliances en netwerk security software maakt en verkoopt.
Deciso is opgericht in 2000 en heeft een sterke focus op open source technologie.

Onze security oplossingen bieden bescherming tegen netwerkbedreigingen in de breedste zin van het woord en vormen de eerste lijn van verdediging tegen malware, trojans, DDOS aanvallen en diverse vormen van fraude.

Onderdeel van de strategie, en broedplaats voor vele technische innovaties, is het snelgroeiende OPNsense open source firewall project. Vanuit dit project zijn technieken rondom high-availability, VPN (beveiligde toegang voor branche offices en ambulante medewerkers), intrusion detection/prevention (door)ontwikkeld en toepasbaar gemaakt voor een breed publiek. De security software wordt onder meer beschikbaar gesteld op een eigen in Nederland ontwikkeld en geproduceerd hardware platform.

Meer weten?
Neem dan contact op met Robert van Papeveld, 0187-744 020

Sollicitatie
We ontvangen je sollicitatie graag op career@deciso.com

Apologies for the delay, had to fix some small formatting issues first.
Docs are now up to date :-)

Thanks to all commiters!

Cheers,

Jos