Hey there,
Since recently we have a new 25Gbit/s (Init7.net) connection with OPNsense.
OPNsense 24.1.8-amd64
FreeBSD 13.2-RELEASE-p11
OpenSSL 3.0.13
Hardware: Minisforums MS-01
CPU: Intel Core i9-13900H
RAM: 32 GB Crucial Soram D5 5200Mhz
Network: Mellanox ConnectX-4 Lx EN 25Gbit SFP28
Storage: Samsung 980 Pro
-----
The throughput is nowhere where it should be, also there is a big packet loss.
I am testing directly from the router and the results are like the following:
Speedtest: root@OPNsense:~ # speedtest -s 43030
Speedtest by Ookla
Server: Init7 AG - Winterthur (id: 43030)
ISP: Init7
Idle Latency: 6.85 ms (jitter: 0.15ms, low: 6.74ms, high: 7.06ms)
Download: 9432.59 Mbps (data used: 10.3 GB)
25.87 ms (jitter: 34.23ms, low: 6.52ms, high: 271.92ms)
Upload: 225.91 Mbps (data used: 168.6 MB)
6.80 ms (jitter: 0.11ms, low: 6.61ms, high: 7.35ms)
Packet Loss: 7.5% Result URL: https://www.speedtest.net/result/c/8c28763f-1d41-4483-9f03-df7b9ec7b9d1
The packet loss is also weird.
iperf3 throws out results such as: Quoteroot@OPNsense:~ # iperf3 -c speedtest.init7.net
Connecting to host speedtest.init7.net, port 5201
[ 5] local <localIP> port 41761 connected to 77.109.175.63 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.06 sec 11.1 MBytes 87.8 Mbits/sec 9 96.6 KBytes
[ 5] 1.06-2.06 sec 9.25 MBytes 77.9 Mbits/sec 6 46.9 KBytes
[ 5] 2.06-3.06 sec 8.12 MBytes 68.1 Mbits/sec 12 46.8 KBytes
[ 5] 3.06-4.06 sec 6.50 MBytes 54.5 Mbits/sec 8 54.0 KBytes
[ 5] 4.06-5.06 sec 7.38 MBytes 61.9 Mbits/sec 8 39.7 KBytes
[ 5] 5.06-6.06 sec 7.38 MBytes 61.9 Mbits/sec 6 62.5 KBytes
[ 5] 6.06-7.06 sec 9.00 MBytes 75.5 Mbits/sec 4 96.7 KBytes
[ 5] 7.06-8.06 sec 8.62 MBytes 72.4 Mbits/sec 6 32.6 KBytes
[ 5] 8.06-9.06 sec 5.38 MBytes 45.1 Mbits/sec 6 72.6 KBytes
[ 5] 9.06-10.06 sec 4.88 MBytes 40.9 Mbits/sec 8 26.9 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.06 sec 77.6 MBytes 64.7 Mbits/sec 73 sender
[ 5] 0.00-10.07 sec 76.8 MBytes 64.0 Mbits/sec receiver
iperf Done.
root@OPNsense:~ #
The more parallel streams I use, the faster it is. If I use 128 parallel streams (with -P, 128 is the maximum), I can get over 7000 Mbits/sec, but nowhere near where it should be.
I have also tried following some tuning guides, such as these here:
https://calomel.org/freebsd_network_tuning.html
https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/
Screenshot of my tunables:
https://drive.proton.me/urls/ZWXZ8C49D0#RucQw3R6Ofhj
Stock settings, enabling RSS, enabling net.isr.bindthreads = 1, net.isr.maxthreads = -1 and some other settings from the guides show sadly no improvement.
Hardware offloading is off (apparently that OPNSense + Mellanox do not work well with that), IDS/IPS is also off. For testing purposed, I also tried hardware offloading on, without differences.
dmesg output for mlx:
root@OPNsense:~ # dmesg
mlx5_core0: <mlx5_core> mem 0x6120000000-0x6121ffffff at device 0.0 on pci1
mlx5: Mellanox Core driver 3.7.1 (November 2021)uhub0: 4 ports with 4 removable, self powered
mlx5_core0: INFO: mlx5_port_module_event:705:(pid 12): Module 0, status: plugged and enabled
mlx5_core: INFO: (mlx5_core0): E-Switch: Total vports 9, l2 table size(65536), per vport: max uc(1024) max mc(16384)
mlx5_core1: <mlx5_core> mem 0x611e000000-0x611fffffff at device 0.1 on pci1
mlx5_core1: INFO: mlx5_port_module_event:710:(pid 12): Module 1, status: unplugged
mlx5_core: INFO: (mlx5_core1): E-Switch: Total vports 9, l2 table size(65536), per vport: max uc(1024) max mc(16384)
mce0: Ethernet address: <mac>
mce0: link state changed to DOWN
mce1: Ethernet address: <mac>
mce1: link state changed to DOWN
mce0: ERR: mlx5e_ioctl:3514:(pid 37363): tso4 disabled due to -txcsum.
mce0: ERR: mlx5e_ioctl:3527:(pid 37959): tso6 disabled due to -txcsum6.
mce1: ERR: mlx5e_ioctl:3514:(pid 41002): tso4 disabled due to -txcsum.
mce1: ERR: mlx5e_ioctl:3527:(pid 41674): tso6 disabled due to -txcsum6.
mce0: INFO: mlx5e_open_locked:3265:(pid 60133): NOTE: There are more RSS buckets(64) than channels(20) available
mce0: link state changed to UP
root@OPNsense:~ # root@OPNsense:~ # ifconfig
mce0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN (wan)
options=7e8800a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO>
ether <mac>
inet <IP> netmask 0xffffffc0 broadcast <broadcast>
inet6 <ip>%mce0 prefixlen 64 scopeid 0x9
inet6 <ip> prefixlen 64 autoconf
inet6 <ip> prefixlen 128
media: Ethernet 25GBase-SR <full-duplex,rxpause,txpause>
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
mce1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=7e8800a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO>
ether <mac>
media: Ethernet autoselect <full-duplex,rxpause,txpause>
status: no carrier (Cable is unplugged.)
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@OPNsense:~ #Here I am a bit surprised about
Ethernet 25GBase-SR, to my limited understanding that should be LR. In OPNsense however I don't see any 25GBase-LR setting to enforce. Autonegotiate will return SR. According to my provider, the SFP is LR:
https://www.init7.net/en/internet/hardware/
Is that just a display error in OPNsense?
Here's a screenshot from Top -P while running a speedtest.https://drive.proton.me/urls/FPZY26VGH4#2oSBskqkz07X
netstat -Qroot@OPNsense:~ # netstat -Q
Configuration:
Setting Current Limit
Thread count 20 20
Default queue limit 2048 10240
Dispatch policy deferred n/a
Threads bound to CPUs enabled n/a
Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 1000 cpu hybrid C--
igmp 2 2048 source default ---
rtsock 3 2048 source default ---
arp 4 2048 source default ---
ether 5 2048 cpu direct C--
ip6 6 1000 cpu hybrid C--
ip_direct 9 2048 cpu hybrid C--
ip6_direct 10 2048 cpu hybrid C--
Workstreams:
WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled
0 0 ip 0 12 0 58203 0 111666 169869
0 0 igmp 0 0 0 0 0 0 0
0 0 rtsock 0 0 0 0 0 0 0
0 0 arp 0 0 0 0 0 0 0
0 0 ether 0 0 2355942 0 0 0 2355942
0 0 ip6 0 1 0 1908712 0 129 1908841
0 0 ip_direct 0 0 0 0 0 0 0
0 0 ip6_direct 0 0 0 0 0 0 0
1 1 ip 0 11 0 27296 0 19571 46867
1 1 igmp 0 0 0 0 0 0 0
1 1 rtsock 0 0 0 0 0 0 0
1 1 arp 0 1 0 0 0 15 15
1 1 ether 0 0 1297838 0 0 0 1297838
1 1 ip6 0 40 0 1269620 0 201 1269821
1 1 ip_direct 0 0 0 0 0 0 0
1 1 ip6_direct 0 0 0 0 0 0 0
2 2 ip 0 13 0 16382 0 17041 33423
2 2 igmp 0 0 0 0 0 0 0
2 2 rtsock 0 0 0 0 0 0 0
2 2 arp 0 0 0 0 0 0 0
2 2 ether 0 0 1501728 0 0 0 1501728
2 2 ip6 0 0 0 1219557 0 0 1219557
2 2 ip_direct 0 0 0 0 0 0 0
2 2 ip6_direct 0 0 0 0 0 0 0
3 3 ip 0 31 0 280587 0 421365 701952
3 3 igmp 0 0 0 0 0 0 0
3 3 rtsock 0 0 0 0 0 0 0
3 3 arp 0 0 0 0 0 0 0
3 3 ether 0 0 1094476 0 0 0 1094476
3 3 ip6 0 0 0 788353 0 0 788353
3 3 ip_direct 0 0 0 0 0 0 0
3 3 ip6_direct 0 0 0 0 0 0 0
4 4 ip 0 10 0 42578 0 5656 48234
4 4 igmp 0 0 0 0 0 0 0
4 4 rtsock 0 0 0 0 0 0 0
4 4 arp 0 0 0 0 0 0 0
4 4 ether 0 0 1803388 0 0 0 1803388
4 4 ip6 0 0 0 1535915 0 0 1535915
4 4 ip_direct 0 0 0 0 0 0 0
4 4 ip6_direct 0 0 0 0 0 0 0
5 5 ip 0 35 0 57631 0 92496 150127
5 5 igmp 0 0 0 0 0 0 0
5 5 rtsock 0 0 0 0 0 0 0
5 5 arp 0 0 0 0 0 0 0
5 5 ether 0 0 1078428 0 0 0 1078428
5 5 ip6 0 1 0 1020797 0 3 1020800
5 5 ip_direct 0 0 0 0 0 0 0
5 5 ip6_direct 0 0 0 0 0 0 0
6 6 ip 0 2 0 27426 0 1041 28467
6 6 igmp 0 0 0 0 0 0 0
6 6 rtsock 0 7 0 0 0 158 158
6 6 arp 0 0 0 0 0 0 0
6 6 ether 0 0 1469570 0 0 0 1469570
6 6 ip6 0 0 0 882669 0 0 882669
6 6 ip_direct 0 0 0 0 0 0 0
6 6 ip6_direct 0 0 0 0 0 0 0
7 7 ip 0 3 0 283352 0 5797 289149
7 7 igmp 0 0 0 0 0 0 0
7 7 rtsock 0 0 0 0 0 0 0
7 7 arp 0 0 0 0 0 0 0
7 7 ether 0 0 1270934 0 0 0 1270934
7 7 ip6 0 0 0 987582 0 0 987582
7 7 ip_direct 0 0 0 0 0 0 0
7 7 ip6_direct 0 0 0 0 0 0 0
8 8 ip 0 28 0 263924 0 55582 319506
8 8 igmp 0 0 0 0 0 0 0
8 8 rtsock 0 0 0 0 0 0 0
8 8 arp 0 0 0 0 0 0 0
8 8 ether 0 0 2150278 0 0 0 2150278
8 8 ip6 0 2 0 1626537 0 68 1626605
8 8 ip_direct 0 0 0 0 0 0 0
8 8 ip6_direct 0 0 0 0 0 0 0
9 9 ip 0 10 0 50414 0 117 50531
9 9 igmp 0 0 0 0 0 0 0
9 9 rtsock 0 0 0 0 0 0 0
9 9 arp 0 2 0 0 0 323528 323528
9 9 ether 0 0 1078713 0 0 0 1078713
9 9 ip6 0 45 0 1027819 0 479 1028298
9 9 ip_direct 0 0 0 0 0 0 0
9 9 ip6_direct 0 0 0 0 0 0 0
10 10 ip 0 7 0 607 0 7940 8547
10 10 igmp 0 0 0 0 0 0 0
10 10 rtsock 0 0 0 0 0 0 0
10 10 arp 0 0 0 0 0 0 0
10 10 ether 0 0 1215919 0 0 0 1215919
10 10 ip6 0 0 0 1201173 0 0 1201173
10 10 ip_direct 0 0 0 0 0 0 0
10 10 ip6_direct 0 0 0 0 0 0 0
11 11 ip 0 58 0 12866 0 210981 223847
11 11 igmp 0 0 0 0 0 0 0
11 11 rtsock 0 0 0 0 0 0 0
11 11 arp 0 3 0 0 0 35 35
11 11 ether 0 0 958646 0 0 0 958646
11 11 ip6 0 0 0 945547 0 0 945547
11 11 ip_direct 0 0 0 0 0 0 0
11 11 ip6_direct 0 0 0 0 0 0 0
12 12 ip 0 8 0 63449 0 44365 107814
12 12 igmp 0 0 0 0 0 0 0
12 12 rtsock 0 0 0 0 0 0 0
12 12 arp 0 0 0 0 0 0 0
12 12 ether 0 0 1513917 0 0 0 1513917
12 12 ip6 0 0 0 1445402 0 0 1445402
12 12 ip_direct 0 0 0 0 0 0 0
12 12 ip6_direct 0 0 0 0 0 0 0
13 13 ip 0 6 0 409033 0 4978 414011
13 13 igmp 0 0 0 0 0 0 0
13 13 rtsock 0 0 0 0 0 0 0
13 13 arp 0 0 0 0 0 0 0
13 13 ether 0 0 1516011 0 0 0 1516011
13 13 ip6 0 90 0 1099859 0 1214 1101073
13 13 ip_direct 0 0 0 0 0 0 0
13 13 ip6_direct 0 0 0 0 0 0 0
14 14 ip 0 14 0 122522 0 29387 151909
14 14 igmp 0 0 0 0 0 0 0
14 14 rtsock 0 0 0 0 0 0 0
14 14 arp 0 0 0 0 0 0 0
14 14 ether 0 0 1011518 0 0 0 1011518
14 14 ip6 0 2 0 887967 0 4 887971
14 14 ip_direct 0 0 0 0 0 0 0
14 14 ip6_direct 0 0 0 0 0 0 0
15 15 ip 0 29 0 1339 0 205348 206687
15 15 igmp 0 0 0 0 0 0 0
15 15 rtsock 0 0 0 0 0 0 0
15 15 arp 0 0 0 0 0 0 0
15 15 ether 0 0 1046188 0 0 0 1046188
15 15 ip6 0 17 0 1001970 0 32 1002002
15 15 ip_direct 0 0 0 0 0 0 0
15 15 ip6_direct 0 0 0 0 0 0 0
16 16 ip 0 7 0 49940 0 219 50159
16 16 igmp 0 0 0 0 0 0 0
16 16 rtsock 0 0 0 0 0 0 0
16 16 arp 0 0 0 0 0 0 0
16 16 ether 0 0 1590175 0 0 0 1590175
16 16 ip6 0 2 0 1490681 0 2 1490683
16 16 ip_direct 0 0 0 0 0 0 0
16 16 ip6_direct 0 0 0 0 0 0 0
17 17 ip 0 51 0 36987 0 244222 281209
17 17 igmp 0 0 0 0 0 0 0
17 17 rtsock 0 0 0 0 0 0 0
17 17 arp 0 0 0 0 0 0 0
17 17 ether 0 0 1168584 0 0 0 1168584
17 17 ip6 0 34 0 1034189 0 158 1034347
17 17 ip_direct 0 0 0 0 0 0 0
17 17 ip6_direct 0 0 0 0 0 0 0
18 18 ip 0 56 0 242236 0 468895 711131
18 18 igmp 0 0 0 0 0 0 0
18 18 rtsock 0 0 0 0 0 0 0
18 18 arp 0 0 0 0 0 0 0
18 18 ether 0 0 1426511 0 0 0 1426511
18 18 ip6 0 0 0 1054665 0 0 1054665
18 18 ip_direct 0 0 0 0 0 0 0
18 18 ip6_direct 0 0 0 0 0 0 0
19 19 ip 0 11 0 38109 0 11074 49183
19 19 igmp 0 0 0 0 0 0 0
19 19 rtsock 0 0 0 0 0 0 0
19 19 arp 0 0 0 0 0 0 0
19 19 ether 0 0 1823691 0 0 0 1823691
19 19 ip6 0 33 0 1587650 0 41 1587691
19 19 ip_direct 0 0 0 0 0 0 0
19 19 ip6_direct 0 0 0 0 0 0 0While I am not new to Linux generally, I am new to Freebsd. Does anyone have some advices or experiences to share? Does anyone use OPNSense with their 25G line or do you have any recommendations?
I do not know what to test anymore.
Thanks in advance!
Observations:
1. The congestion window seems way too small to reach higher speeds. This also correlates with the high interrupt load on all CPUs. IDK how to enlarge buffers for this card, maybe there are specific parameters to be set for that.
2. The MS-01 only has a PCIe 4.0x4 slot. You NIC is limited to PCIe 3.0x8, so in effect you will be getting PCIe 3.0x4 at most. This should suffice for ~32 Gbit/s, however, potentially the card does not handle a limited PCIe bus width too well.
1) Thank you. Where do you see that? Then I can look into it?
2) Regarding this point, while I am certainly no expert here, all information I can find is that the MS-01 has as PCIe 4.0 x16 slot, supporting PCIe 4.0 x8 (128GT/s)
https://store.minisforum.com/products/minisforum-ms-01
https://store.minisforum.de/products/ms-01
https://www.techradar.com/computing/hands-on-minisforum-ms-01-review
That's why I don't understand this point too well :P
w/r to 1): The Cwnd window is only 100 KBytes max. When I do the same on an Intel X520 card, I get almost 1 GBbyte Cwnd.
w/r to 2): Yes, I got it wrong. The slot is PCIe 4.0x8. I had the Ugreen NaSync series in mind... Sorry.
Quote from: 80ec110286 on June 13, 2024, 11:39:02 AM
...
Does anyone use OPNSense with their 25G line or do you have any recommendations?
...
I'm not a Mellanox expert, but you might want to look at your NIC firmware settings (are you running latest firmware update ?) regarding the operational mode: Native NIC mode vs Switchdev SRIOV.
When doing benchmarks be 100% sure you're in Native NIC mode.
mlx5_core: INFO: (mlx5_core0): E-Switch: Total vports 9, l2 table size(65536), per vport: max uc(1024) max mc(16384)When looking for throughput with >10Gb Ethernet, the first quick win is using Jumbo's (MTU 9000), although this won't benefit your WAN connection (PMTUD).
mce0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
QuoteI'm not a Mellanox expert, but you might want to look at your NIC firmware settings (are you running latest firmware update ?)
The latest firmware as is possible.. which is december 2021 :-P
Quoteregarding the operational mode: Native NIC mode vs Switchdev SRIOV.
I tried disabling SR-IOV in Opnsense, no difference. Should I have it disabled in the BIOS as well?
Quote from: 80ec110286 on June 18, 2024, 04:56:11 PM
I tried disabling SR-IOV in Opnsense, no difference. Should I have it disabled in the BIOS as well?
It's worth a try...
Last three months I experienced major issues with a SuperMicro AM5 board and a 25Gb Intel 810 ethernet adapter, after endless debugging, two interim BIOS fixes and a new firmware update for the Intel 810 it looks like things are now mostly fixed, at least the "forced" SR-IOV mode I experienced and some other stuff that both machine and NIC BIOS/Firmware should have done but didn't.
You confirmed that the MS-01 is using PCie 4 x8, which should be sufficient for your card. I couldn't find a chipset diagram for this system, but be sure these are dedicated lanes directly connected to your CPU, not hidden behind some internal (shared) mux or chipset.
Quote from: netnut on June 18, 2024, 08:29:58 PM
Last three months I experienced major issues with a SuperMicro AM5 board and a 25Gb Intel 810 ethernet adapter, after endless debugging, two interim BIOS fixes and a new firmware update for the Intel 810 it looks like things are now mostly fixed, at least the "forced" SR-IOV mode I experienced and some other stuff that both machine and NIC BIOS/Firmware should have done but didn't.
Did you set ice_ddp_load="YES"' in /boot/loader.conf.local? Otherwise OPNsense will only use only one queue on the NIC, which limits packet processing to one core, which in most cases is not enough for 25GBit throughput.
QuoteDid you set ice_ddp_load="YES"' in /boot/loader.conf.local? Otherwise OPNsense will only use only one queue on the NIC, which limits packet processing to one core, which in most cases is not enough for 25GBit throughput.
I am curious whether there's something similar for Mellanox cards.
Do you have any update? I am planning on getting the same hardware. I also wonder about the heat, have you experienced a high increase in heat with the mellanox card?
Btw have you enabled the drivers?
https://www.thomas-krenn.com/de/wiki/OPNsense_Chelsio_Mellanox_Broadcom_Netzwerkkarten-Treiber_aktivieren