Hi,
I have just upgraded two freshly installed OPNsense 24.7.7 instances (connected to HA with CARP IPs, etc.) to 24.7.8, and after that, my network stopped working.
Network configuration:
WAN: 2x 25Gbps -> lagg0 -> lagg0_vlan52
LAN: 2x 25Gbps -> lagg0 -> lagg0_vlan5
pfsync: 2x 25Gbps -> lagg0 -> lagg0_vlan2
bnxt0 and bnxt1 are up/lower_up
status: active
lagg0 is up/lower_up but:
laggport: bnxt0 flags=0<>
laggport: bnxt1 flags=0<>
status: active
lagg0_vlan2 is up/lower_up
vlan: 2
vlanproto: 802.1q
vlanpcp: 0
parent interface: lagg0
groups: vlan
status: active
However, no network connection is working. I can't even ping the backup host on any interface.
When I reboot my server, I see many instances of the following error:
HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error
Did I do something wrong?
Regards
Borys O
There is a problem with that line of NICs which is being discussed at least since January 2023:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133
You probably best get a different brand of cards.
EDIT: possibly a different firmware version might help:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245981
Is there any chance to "undo" upgrade?
If you created a ZFS snapshot beforehand, yes. If you did not, you will have to reinstall.
Unsure how this is directly related to 24.7.8 as bnxt driver wasn't touched. Testing the older kernel makes sense, but only for diagnosis:
# opnsene-update -kr 24.7.6
Cheers,
Franco
It's strange that the network works after shutdown for couple minutes. At the moment, one of the machines is working (the network is functional)."
root@admins-opn-1:~ # uname -a
FreeBSD admins-opn-1.redgelabs.com 14.1-RELEASE-p6 FreeBSD 14.1-RELEASE-p6 stable/24.7-n267939-fd5bc7f34e1 SMP amd64
root@admins-opn-1:~ # opnsense-version
OPNsense 24.7.8
root@admins-opn-1:~ # uptime
12:47PM up 7 mins, 2 users, load averages: 0.11, 0.12, 0.07
root@admins-opn-1:~ #
root@admins-opn-1:~ # ifconfig lagg0
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether 7c:c2:55:99:66:8e
hwaddr 00:00:00:00:00:00
inet6 fe80::7ec2:55ff:fe99:668e%lagg0 prefixlen 64 scopeid 0x9
laggproto lacp lagghash l2,l3
laggport: bnxt0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: bnxt1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
root@admins-opn-1:~ # pciconf -l -BbcevV bnxt0
bnxt0@pci0:1:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x14e4 device=0x16d7 subvendor=0x15d9 subdevice=0x0928
vendor = 'Broadcom Inc. and subsidiaries'
device = 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0xdfe10000, size 65536, enabled
bar [18] = type Prefetchable Memory, range 64, base 0xdfd00000, size 1048576, enabled
bar [20] = type Prefetchable Memory, range 64, base 0xdfe22000, size 8192, enabled
cap 01[48] = powerspec 3 supports D0 D3 current D0
cap 03[50] = VPD
cap 11[a0] = MSI-X supports 74 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x4a0]
cap 10[ac] = PCI-Express 2 endpoint max data 256(512) FLR NS
max read 512
link x8(x8) speed 8.0(8.0) ClockPM disabled
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[13c] = Serial 1 7cc255fffe99668e
ecap 0004[150] = Power Budgeting 1
ecap 0002[160] = VC 1 max VC0
ecap 000b[180] = Vendor [1] ID 0000 Rev 0 Length 32
ecap 0018[1b0] = LTR 1
ecap 000e[1b8] = ARI 1
ecap 0017[230] = TPH Requester 1
ecap 0019[300] = PCIe Sec 1 lane errors 0
ecap 001f[200] = Precision Time Measurement 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
VPD ident = 'Supermicro Network Adapter'
VPD ro PN = 'AOC-S25G-b2S'
VPD ro SN = 'XXX'
VPD ro V0 = '228.0.116.0'
VPD ro V1 = '1.02'
VPD ro V2 = '7CC25599668E'
VPD ro V3 = '7CC25599668F'
VPD ro V4 = '000000000000'
VPD ro V5 = '000000000000'
VPD ro VA = '2'
root@admins-opn-1:~ # pciconf -l -BbcevV bnxt1
bnxt1@pci0:1:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x14e4 device=0x16d7 subvendor=0x15d9 subdevice=0x0928
vendor = 'Broadcom Inc. and subsidiaries'
device = 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0xdfe00000, size 65536, enabled
bar [18] = type Prefetchable Memory, range 64, base 0xdfc00000, size 1048576, enabled
bar [20] = type Prefetchable Memory, range 64, base 0xdfe20000, size 8192, enabled
cap 01[48] = powerspec 3 supports D0 D3 current D0
cap 03[50] = VPD
cap 11[a0] = MSI-X supports 74 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x4a0]
cap 10[ac] = PCI-Express 2 endpoint max data 256(512) FLR NS
max read 512
link x8(x8) speed 8.0(8.0) ClockPM disabled
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[13c] = Serial 1 7cc255fffe99668e
ecap 0004[150] = Power Budgeting 1
ecap 000b[180] = Vendor [1] ID 0000 Rev 0 Length 32
ecap 000e[1b8] = ARI 1
ecap 0017[230] = TPH Requester 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
VPD ident = 'Supermicro Network Adapter'
VPD ro PN = 'AOC-S25G-b2S'
VPD ro SN = 'XXX'
VPD ro V0 = '228.0.116.0'
VPD ro V1 = '1.02'
VPD ro V2 = '7CC25599668E'
VPD ro V3 = '7CC25599668F'
VPD ro V4 = '000000000000'
VPD ro V5 = '000000000000'
VPD ro VA = '2'
Quote from: Patrick M. Hausen on November 12, 2024, 09:40:59 AM
There is a problem with that line of NICs which is being discussed at least since January 2023:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133
You probably best get a different brand of cards.
EDIT: possibly a different firmware version might help:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245981
Hi
After changing NIC to Intel XXV710 I still have network problems.
server1:
Name
admins-opn-1
Versions
OPNsense 24.7.10_1-amd64
FreeBSD 14.1-RELEASE-p6
OpenSSL 3.0.15
root@admins-opn-1:~ # pciconf -l -BbcevV ixl0
ixl0@pci0:1:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x158b subvendor=0x15d9 subdevice=0x0978
vendor = 'Intel Corporation'
device = 'Ethernet Controller XXV710 for 25GbE SFP28'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0xdc000000, size 16777216, enabled
bar [1c] = type Prefetchable Memory, range 64, base 0xdd008000, size 32768, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 129 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR
max read 512
link x8(x8) speed 8.0(8.0)
cap 03[e0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x0110, VF RID Stride 0x0001
VF Device ID 0x154c
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
ecap 0019[1d0] = PCIe Sec 1 lane errors 0xff
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
VPD ident = 'Supermicro Network Adapter'
VPD ro PN = 'AOC-S25G-i2S '
VPD ro V0 = '9.30 0x8000EA8A'
VPD ro V1 = '2.00 '
VPD ro SN = ' '
VPD ro VA = '2'
VPD ro V2 = ''
VPD ro V3 = ''
VPD ro V4 = ' '
VPD ro V5 = ' '
VPD rw VB = ''
root@admins-opn-1:~ # pciconf -l -BbcevV ixl1
ixl1@pci0:1:0:1: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x158b subvendor=0x15d9 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'Ethernet Controller XXV710 for 25GbE SFP28'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0xdb000000, size 16777216, enabled
bar [1c] = type Prefetchable Memory, range 64, base 0xdd000000, size 32768, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 129 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR
max read 512
link x8(x8) speed 8.0(8.0)
cap 03[e0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x014f, VF RID Stride 0x0001
VF Device ID 0x154c
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
VPD ident = 'Supermicro Network Adapter'
VPD ro PN = 'AOC-S25G-i2S '
VPD ro V0 = '9.30 0x8000EA8A'
VPD ro V1 = '2.00 '
VPD ro SN = ' '
VPD ro VA = '2'
VPD ro V2 = ''
VPD ro V3 = ''
VPD ro V4 = ' '
VPD ro V5 = ' '
VPD rw VB = ''
root@admins-opn-1:~ #
root@admins-opn-1:~ #
root@admins-opn-1:~ # ifconfig lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether xx:xx:xx:xx:11:b4
hwaddr 00:00:00:00:00:00
laggproto lacp lagghash l2,l3
laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
root@admins-opn-1:~ # ifconfig lagg0_vlan52
lagg0_vlan52: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
description: WAN (wan)
options=4000000<MEXTPG>
ether xx:xx:xx:xx:11:b4
inet 192.168.1.1 netmask 0xffffffe0 broadcast 192.168.1.31
groups: vlan
vlan: 52 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
root@admins-opn-1:~ # kldstat
Id Refs Address Size Name
1 61 0xffffffff80200000 1f63788 kernel
2 1 0xffffffff82164000 1e2c8 opensolaris.ko
3 1 0xffffffff82183000 165f0 if_lagg.ko
4 2 0xffffffff8219a000 3540 if_infiniband.ko
5 1 0xffffffff8219f000 ea58 if_bridge.ko
6 2 0xffffffff821ae000 8930 bridgestp.ko
7 1 0xffffffff821b7000 3c10 pflog.ko
8 3 0xffffffff821bb000 8d578 pf.ko
9 1 0xffffffff82249000 11a18 pfsync.ko
10 1 0xffffffff8225b000 5cd5e0 zfs.ko
11 1 0xffffffff82829000 aa30 if_gre.ko
12 1 0xffffffff82834000 4be0 if_enc.ko
13 1 0xffffffff82839000 fba8 carp.ko
14 1 0xffffffff83220000 2110 pchtherm.ko
15 1 0xffffffff83223000 3250 ichsmb.ko
16 1 0xffffffff83227000 2178 smbus.ko
17 1 0xffffffff8322a000 3360 uhid.ko
18 1 0xffffffff8322e000 4364 ums.ko
19 1 0xffffffff83233000 33c0 usbhid.ko
20 1 0xffffffff83237000 3380 hidbus.ko
21 1 0xffffffff8323b000 4850 nullfs.ko
root@admins-opn-1:~ #
server2:Name
admins-opn-2
Versions
OPNsense 24.7.10_1-amd64
FreeBSD 14.1-RELEASE-p6
OpenSSL 3.0.15
root@admins-opn-2:~ # pciconf -l -BbcevV ixl0
ixl0@pci0:1:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x158b subvendor=0x15d9 subdevice=0x0978
vendor = 'Intel Corporation'
device = 'Ethernet Controller XXV710 for 25GbE SFP28'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0xdc000000, size 16777216, enabled
bar [1c] = type Prefetchable Memory, range 64, base 0xdd008000, size 32768, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 129 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR
max read 512
link x8(x8) speed 8.0(8.0)
cap 03[e0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x0110, VF RID Stride 0x0001
VF Device ID 0x154c
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
ecap 0019[1d0] = PCIe Sec 1 lane errors 0xff
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
VPD ident = 'Supermicro Network Adapter'
VPD ro PN = 'AOC-S25G-i2S '
VPD ro V0 = '9.30 0x8000EA8A'
VPD ro V1 = '2.00 '
VPD ro SN = ' '
VPD ro VA = '2'
VPD ro V2 = ''
VPD ro V3 = ''
VPD ro V4 = ' '
VPD ro V5 = ' '
VPD rw VB = ''
root@admins-opn-2:~ # pciconf -l -BbcevV ixl1
ixl1@pci0:1:0:1: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x158b subvendor=0x15d9 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'Ethernet Controller XXV710 for 25GbE SFP28'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0xdb000000, size 16777216, enabled
bar [1c] = type Prefetchable Memory, range 64, base 0xdd000000, size 32768, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 129 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR
max read 512
link x8(x8) speed 8.0(8.0)
cap 03[e0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x014f, VF RID Stride 0x0001
VF Device ID 0x154c
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
VPD ident = 'Supermicro Network Adapter'
VPD ro PN = 'AOC-S25G-i2S '
VPD ro V0 = '9.30 0x8000EA8A'
VPD ro V1 = '2.00 '
VPD ro SN = ' '
VPD ro VA = '2'
VPD ro V2 = ''
VPD ro V3 = ''
VPD ro V4 = ' '
VPD ro V5 = ' '
VPD rw VB = ''
root@admins-opn-2:~ # ifconfig lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether yy:yy:yy:yy:17:ba
hwaddr 00:00:00:00:00:00
laggproto lacp lagghash l2,l3
laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
root@admins-opn-2:~ # ifconfig lagg0_vlan52
lagg0_vlan52: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
description: WAN (wan)
options=4000000<MEXTPG>
ether yy:yy:yy:yy:17:ba
inet 192.168.1.2 netmask 0xffffffe0 broadcast 192.168.1.31
groups: vlan
vlan: 52 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
root@admins-opn-2:~ # kldstat
Id Refs Address Size Name
1 61 0xffffffff80200000 1f63788 kernel
2 1 0xffffffff82164000 4be0 if_enc.ko
3 1 0xffffffff82169000 1e2c8 opensolaris.ko
4 1 0xffffffff82188000 165f0 if_lagg.ko
5 2 0xffffffff8219f000 3540 if_infiniband.ko
6 1 0xffffffff821a3000 ea58 if_bridge.ko
7 2 0xffffffff821b2000 8930 bridgestp.ko
8 1 0xffffffff821bb000 5cd5e0 zfs.ko
9 1 0xffffffff82789000 11a18 pfsync.ko
10 3 0xffffffff8279b000 8d578 pf.ko
11 1 0xffffffff82829000 fba8 carp.ko
12 1 0xffffffff82839000 3c10 pflog.ko
13 1 0xffffffff8283d000 aa30 if_gre.ko
14 1 0xffffffff83220000 2110 pchtherm.ko
15 1 0xffffffff83223000 3250 ichsmb.ko
16 1 0xffffffff83227000 2178 smbus.ko
17 1 0xffffffff8322a000 3360 uhid.ko
18 1 0xffffffff8322e000 4364 ums.ko
19 1 0xffffffff83233000 33c0 usbhid.ko
20 1 0xffffffff83237000 3380 hidbus.ko
21 1 0xffffffff8323b000 4850 nullfs.ko
root@admins-opn-2:~ #
iperf3 test result between this 2 server:server1:root@admins-opn-1:~ # pfctl -d
pfctl: pf not enabled
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 2124 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.04 sec 0.00 Bytes 0.00 bits/sec 3 8.95 KBytes
[ 5] 1.04-2.05 sec 0.00 Bytes 0.00 bits/sec 2 8.95 KBytes
[ 5] 2.05-3.06 sec 0.00 Bytes 0.00 bits/sec 0 8.95 KBytes
[ 5] 3.06-4.06 sec 0.00 Bytes 0.00 bits/sec 1 8.95 KBytes
[ 5] 4.06-5.00 sec 0.00 Bytes 0.00 bits/sec 0 8.95 KBytes
[ 5] 5.00-6.06 sec 0.00 Bytes 0.00 bits/sec 1 8.95 KBytes
[ 5] 6.06-7.04 sec 0.00 Bytes 0.00 bits/sec 0 8.95 KBytes
[ 5] 7.04-8.04 sec 0.00 Bytes 0.00 bits/sec 0 8.95 KBytes
[ 5] 8.04-9.04 sec 0.00 Bytes 0.00 bits/sec 0 8.95 KBytes
[ 5] 9.04-10.04 sec 0.00 Bytes 0.00 bits/sec 1 8.95 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec 8 sender
[ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec receiver
iperf Done.
server2:root@admins-opn-2:~ # pfctl -d
pfctl: pf not enabled
root@admins-opn-2:~ # iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.1.1, port 17875
[ 5] local 192.168.1.2 port 5201 connected to 192.168.1.1 port 2124
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 1.06-2.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 2.06-3.02 sec 0.00 Bytes 0.00 bits/sec
[ 5] 3.02-4.03 sec 0.00 Bytes 0.00 bits/sec
[ 5] 4.03-5.05 sec 0.00 Bytes 0.00 bits/sec
[ 5] 5.05-6.03 sec 0.00 Bytes 0.00 bits/sec
[ 5] 6.03-7.05 sec 0.00 Bytes 0.00 bits/sec
[ 5] 7.05-8.04 sec 0.00 Bytes 0.00 bits/sec
[ 5] 8.04-9.06 sec 0.00 Bytes 0.00 bits/sec
[ 5] 9.06-10.04 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------On this hardware under Ubuntu 22.04 LTS (installed on one of disks) with same lag+vlan+ip addresses, iperf3 achieves about 23Gbps without any problems (look at attached screens).
Am I doing something wrong?
Tomorrow I will "disassemble" the lags and do a test on the vlan only ports. After that on full access port.
Any help/suggestion will be appreciated.
Regards
Borys
Small update:
Removed lag and vlan configuration. Currently WAN port is "access" port.
System up to date.
root@admins-opn-1:~ # uname -a
FreeBSD admins-opn-1.redgelabs.com 14.1-RELEASE-p6 FreeBSD 14.1-RELEASE-p6 stable/24.7-n267979-0d692990122 SMP amd64
root@admins-opn-1:~ # sysctl dev.ixl.0.fw_lldp=0
dev.ixl.0.fw_lldp: 1 -> 0
root@admins-opn-1:~ # sysctl dev.ixl.1.fw_lldp=0
dev.ixl.1.fw_lldp: 1 -> 0
root@admins-opn-1:~ # sysctl -a | grep bindthreads
net.isr.bindthreads: 1
root@admins-opn-1:~ # ifconfig ixl0
ixl0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether xx:xx:xx:xx:11:b4
inet 192.168.1.1 netmask 0xffffffe0 broadcast 192.168.1.31
media: Ethernet autoselect (25GBase-LR <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@admins-opn-1:~ #
root@admins-opn-1:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.1.%desc: Intel(R) I210 (Copper)
dev.igb.0.%desc: Intel(R) I210 (Copper)
dev.ixl.1.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k
dev.ixl.0.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k
root@admins-opn-1:~ #
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 61382 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.06 sec 313 MBytes 2.47 Gbits/sec 413 112 KBytes
[ 5] 1.06-2.00 sec 430 MBytes 3.83 Gbits/sec 0 723 KBytes
[ 5] 2.00-3.05 sec 477 MBytes 3.82 Gbits/sec 0 1.05 MBytes
[ 5] 3.05-4.06 sec 463 MBytes 3.85 Gbits/sec 0 1.05 MBytes
[ 5] 4.06-5.03 sec 441 MBytes 3.82 Gbits/sec 0 1.05 MBytes
[ 5] 5.03-6.00 sec 444 MBytes 3.83 Gbits/sec 0 1.05 MBytes
[ 5] 6.00-7.00 sec 457 MBytes 3.83 Gbits/sec 0 1.05 MBytes
[ 5] 7.00-8.02 sec 464 MBytes 3.81 Gbits/sec 0 1.05 MBytes
[ 5] 8.02-9.00 sec 448 MBytes 3.84 Gbits/sec 0 1.05 MBytes
[ 5] 9.00-10.00 sec 458 MBytes 3.84 Gbits/sec 0 1.05 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.29 GBytes 3.69 Gbits/sec 413 sender
[ 5] 0.00-10.00 sec 4.29 GBytes 3.69 Gbits/sec receiver
iperf Done.
After adding Intels newest driver:
root@admins-opn-1:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.1.%desc: Intel(R) I210 (Copper)
dev.igb.0.%desc: Intel(R) I210 (Copper)
dev.ixl.1.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 43611 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 432 MBytes 3.63 Gbits/sec 94 931 KBytes
[ 5] 1.00-2.00 sec 430 MBytes 3.61 Gbits/sec 0 931 KBytes
[ 5] 2.00-3.01 sec 433 MBytes 3.61 Gbits/sec 0 931 KBytes
[ 5] 3.01-4.00 sec 427 MBytes 3.60 Gbits/sec 0 931 KBytes
[ 5] 4.00-5.05 sec 450 MBytes 3.61 Gbits/sec 0 931 KBytes
[ 5] 5.05-6.05 sec 429 MBytes 3.60 Gbits/sec 0 931 KBytes
[ 5] 6.05-7.00 sec 411 MBytes 3.61 Gbits/sec 0 931 KBytes
[ 5] 7.00-8.00 sec 431 MBytes 3.61 Gbits/sec 0 931 KBytes
[ 5] 8.00-9.00 sec 431 MBytes 3.62 Gbits/sec 0 931 KBytes
[ 5] 9.00-10.05 sec 459 MBytes 3.67 Gbits/sec 0 931 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.05 sec 4.23 GBytes 3.62 Gbits/sec 94 sender
[ 5] 0.00-10.05 sec 4.23 GBytes 3.61 Gbits/sec receiver
iperf Done.
Enable LLDP:
root@admins-opn-1:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.1.%desc: Intel(R) I210 (Copper)
dev.igb.0.%desc: Intel(R) I210 (Copper)
dev.ixl.1.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2
root@admins-opn-1:~ # sysctl -a|grep lldp
dev.ixl.1.fw_lldp: 0
dev.ixl.0.fw_lldp: 0
root@admins-opn-1:~ # sysctl dev.ixl.0.fw_lldp=1
dev.ixl.0.fw_lldp: 0 -> 1
root@admins-opn-1:~ # sysctl dev.ixl.1.fw_lldp=1
dev.ixl.1.fw_lldp: 0 -> 1
root@admins-opn-1:~ # sysctl -a | grep lldp
dev.ixl.1.fw_lldp: 1
dev.ixl.0.fw_lldp: 1
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 30248 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 422 MBytes 3.53 Gbits/sec 0 1.08 MBytes
[ 5] 1.00-2.00 sec 411 MBytes 3.45 Gbits/sec 0 1.08 MBytes
[ 5] 2.00-3.00 sec 413 MBytes 3.46 Gbits/sec 0 1.08 MBytes
[ 5] 3.00-4.00 sec 414 MBytes 3.47 Gbits/sec 0 1.08 MBytes
[ 5] 4.00-5.00 sec 413 MBytes 3.46 Gbits/sec 0 1.08 MBytes
[ 5] 5.00-6.00 sec 414 MBytes 3.47 Gbits/sec 0 1.08 MBytes
[ 5] 6.00-7.01 sec 418 MBytes 3.47 Gbits/sec 0 1.08 MBytes
[ 5] 7.01-8.00 sec 410 MBytes 3.48 Gbits/sec 0 1.18 MBytes
[ 5] 8.00-9.00 sec 413 MBytes 3.47 Gbits/sec 0 1.18 MBytes
[ 5] 9.00-10.00 sec 413 MBytes 3.47 Gbits/sec 0 1.18 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec receiver
iperf Done.
Firmware and NVM version:
ixl0: fw 9.130.73618 api 1.15 nvm 9.30 etid 8000ea8a oem 1.269.0
Speedtest-cli:
root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXXX (192.168.1.1)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Exatel S.A (Warsaw): 9.006 ms
Download: 3924.67 Mbit/s
Upload: 1042.79 Mbit/s
Currently waiting for older fw from Supermicro to do more tests.
Regards
Borys
Tough sh*t and I honestly have no idea at the moment. Your HW is quite top notch, so small chance to find anyone with more experience than you and I and that particular hardware. Sorry.
I was managed to get Supermiro NIC fw 8.30, 8.50, 9.00 so I will do more tests.
FW: 9.00 default OPNsense ixl driver:
dev.ixl.0.fw_lldp: 0
dev.ixl.0.fw_version: fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d298 oem 1.268.0
dev.ixl.0.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k
root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXX (192.168.1.1)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 1.434 ms
Testing download speed................................................................................
Download: 4068.19 Mbit/s
Testing upload speed...
Upload: 1141.39 Mbit/s
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 54613 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.02 sec 468 MBytes 3.86 Gbits/sec 25 846 KBytes
[ 5] 1.02-2.06 sec 489 MBytes 3.93 Gbits/sec 0 846 KBytes
[ 5] 2.06-3.00 sec 439 MBytes 3.93 Gbits/sec 0 857 KBytes
[ 5] 3.00-4.04 sec 482 MBytes 3.91 Gbits/sec 0 857 KBytes
[ 5] 4.04-5.06 sec 478 MBytes 3.91 Gbits/sec 0 857 KBytes
[ 5] 5.06-6.01 sec 446 MBytes 3.92 Gbits/sec 0 857 KBytes
[ 5] 6.01-7.01 sec 468 MBytes 3.94 Gbits/sec 0 857 KBytes
[ 5] 7.01-8.02 sec 471 MBytes 3.92 Gbits/sec 0 857 KBytes
[ 5] 8.02-9.02 sec 469 MBytes 3.92 Gbits/sec 0 857 KBytes
[ 5] 9.02-10.00 sec 460 MBytes 3.92 Gbits/sec 0 857 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.56 GBytes 3.91 Gbits/sec 25 sender
[ 5] 0.00-10.00 sec 4.56 GBytes 3.91 Gbits/sec receiver
iperf Done.
FW: 9.00 Intels 1.42.2 ixl driver:
dev.ixl.0.fw_lldp: 0
dev.ixl.0.fw_version: fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d298 oem 1.268.0
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2
root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXX (192.168.1.1)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 1.274 ms
Testing download speed................................................................................
Download: 3682.38 Mbit/s
Testing upload speed...
Upload: 1488.75 Mbit/s
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 34854 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 414 MBytes 3.47 Gbits/sec 57 1.06 MBytes
[ 5] 1.00-2.00 sec 416 MBytes 3.49 Gbits/sec 0 1.10 MBytes
[ 5] 2.00-3.00 sec 422 MBytes 3.54 Gbits/sec 0 1.10 MBytes
[ 5] 3.00-4.03 sec 436 MBytes 3.54 Gbits/sec 0 1.19 MBytes
[ 5] 4.03-5.00 sec 409 MBytes 3.55 Gbits/sec 0 1.19 MBytes
[ 5] 5.00-6.04 sec 435 MBytes 3.52 Gbits/sec 0 1.19 MBytes
[ 5] 6.04-7.00 sec 407 MBytes 3.54 Gbits/sec 0 1.19 MBytes
[ 5] 7.00-8.00 sec 420 MBytes 3.52 Gbits/sec 0 1.19 MBytes
[ 5] 8.00-9.00 sec 425 MBytes 3.57 Gbits/sec 0 1.19 MBytes
[ 5] 9.00-10.00 sec 424 MBytes 3.55 Gbits/sec 0 1.19 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.11 GBytes 3.53 Gbits/sec 57 sender
[ 5] 0.00-10.00 sec 4.11 GBytes 3.53 Gbits/sec receiver
iperf Done.
FW: 8.50 default OPNsense ixl driver:
dev.ixl.0.fw_version: fw 8.5.67516 api 1.15 nvm 8.50 etid 8000b8b2 oem 1.268.0
dev.ixl.0.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k
Errors appeared in dmesg:
ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_ENOENT
ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_ENOENT
ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_ENOENT
ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_ENOENT
ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_ENOENT
root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXX (192.168.1.1)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 1.273 ms
Download: 4036.40 Mbit/s
Upload: 1165.60 Mbit/s
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 30652 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 411 MBytes 3.44 Gbits/sec 13 1.04 MBytes
[ 5] 1.00-2.05 sec 432 MBytes 3.46 Gbits/sec 0 1.05 MBytes
[ 5] 2.05-3.02 sec 400 MBytes 3.45 Gbits/sec 0 1.11 MBytes
[ 5] 3.02-4.00 sec 406 MBytes 3.46 Gbits/sec 0 1.11 MBytes
[ 5] 4.00-5.06 sec 438 MBytes 3.46 Gbits/sec 0 1.11 MBytes
[ 5] 5.06-6.04 sec 401 MBytes 3.46 Gbits/sec 0 1.11 MBytes
[ 5] 6.04-7.05 sec 421 MBytes 3.47 Gbits/sec 0 1.19 MBytes
[ 5] 7.05-8.00 sec 393 MBytes 3.46 Gbits/sec 0 1.19 MBytes
[ 5] 8.00-9.00 sec 410 MBytes 3.45 Gbits/sec 0 1.19 MBytes
[ 5] 9.00-10.04 sec 429 MBytes 3.46 Gbits/sec 0 1.19 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 4.04 GBytes 3.46 Gbits/sec 13 sender
[ 5] 0.00-10.04 sec 4.04 GBytes 3.46 Gbits/sec receiver
iperf Done.
FW: 8.50 Intels 1.42.2 ixl driver:
dev.ixl.0.fw_version: fw 8.5.67516 api 1.15 nvm 8.50 etid 8000b8b2 oem 1.268.0
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2
No errors in dmesg:
root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXX (192.168.1.1)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 1.36 ms
Testing download speed................................................................................
Download: 3593.34 Mbit/s
Testing upload speed...
Upload: 1441.83 Mbit/s
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 12016 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.06 sec 405 MBytes 3.20 Gbits/sec 55 920 KBytes
[ 5] 1.06-2.00 sec 356 MBytes 3.17 Gbits/sec 0 980 KBytes
[ 5] 2.00-3.06 sec 400 MBytes 3.16 Gbits/sec 0 980 KBytes
[ 5] 3.06-4.04 sec 368 MBytes 3.16 Gbits/sec 0 989 KBytes
[ 5] 4.04-5.00 sec 361 MBytes 3.16 Gbits/sec 0 1020 KBytes
[ 5] 5.00-6.00 sec 377 MBytes 3.16 Gbits/sec 0 1020 KBytes
[ 5] 6.00-7.02 sec 385 MBytes 3.17 Gbits/sec 0 1020 KBytes
[ 5] 7.02-8.00 sec 368 MBytes 3.15 Gbits/sec 0 1020 KBytes
[ 5] 8.00-9.06 sec 400 MBytes 3.17 Gbits/sec 0 1020 KBytes
[ 5] 9.06-10.04 sec 370 MBytes 3.16 Gbits/sec 0 1020 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 3.70 GBytes 3.16 Gbits/sec 55 sender
[ 5] 0.00-10.04 sec 3.70 GBytes 3.16 Gbits/sec receiver
iperf Done.
iperf3 to Ubuntu serwer with 10Gbps NIC:
root@admins-opn-1:~ # iperf3 -c 192.168.1.3
Connecting to host 192.168.1.3, port 5201
[ 5] local 192.168.1.1 port 44011 connected to 192.168.1.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 742 MBytes 6.22 Gbits/sec 45 268 KBytes
[ 5] 1.00-2.00 sec 722 MBytes 6.06 Gbits/sec 55 240 KBytes
[ 5] 2.00-3.00 sec 714 MBytes 5.98 Gbits/sec 110 157 KBytes
[ 5] 3.00-4.00 sec 714 MBytes 6.00 Gbits/sec 77 66.9 KBytes
[ 5] 4.00-5.02 sec 712 MBytes 5.86 Gbits/sec 101 339 KBytes
[ 5] 5.02-6.05 sec 743 MBytes 6.03 Gbits/sec 80 147 KBytes
[ 5] 6.05-7.04 sec 707 MBytes 6.05 Gbits/sec 66 377 KBytes
[ 5] 7.04-8.00 sec 704 MBytes 6.12 Gbits/sec 53 283 KBytes
[ 5] 8.00-9.00 sec 738 MBytes 6.17 Gbits/sec 55 318 KBytes
[ 5] 9.00-10.00 sec 726 MBytes 6.08 Gbits/sec 42 255 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 7.06 GBytes 6.06 Gbits/sec 684 sender
[ 5] 0.00-10.01 sec 7.06 GBytes 6.06 Gbits/sec receiver
iperf Done.
@franco this has to be something with OPNsense just look:
Clean FreeBSD 14.1 install (on the same hardware - third disk :) ):
# uname -a
FreeBSD admins-opn-1 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64FW: 8.50 and default FreeBSD ixl driver:
dev.ixl.0.fw_version: fw 8.5.67516 api 1.15 nvm 8.50 etid 8000b8b2 oem 1.268.0
dev.ixl.0.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k
# speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXX (x.x.x.x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 2.695 ms
Testing download speed................................................................................
Download: 6703.99 Mbit/s
Testing upload speed...
Upload: 2069.06 Mbit/s
iperf3 to OPNsense up to date server:# iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 34871 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.06 sec 436 MBytes 3.44 Gbits/sec 67 883 KBytes
[ 5] 1.06-2.06 sec 408 MBytes 3.44 Gbits/sec 0 943 KBytes
[ 5] 2.06-3.06 sec 411 MBytes 3.45 Gbits/sec 0 943 KBytes
[ 5] 3.06-4.06 sec 411 MBytes 3.45 Gbits/sec 0 943 KBytes
[ 5] 4.06-5.01 sec 390 MBytes 3.45 Gbits/sec 0 943 KBytes
[ 5] 5.01-6.03 sec 420 MBytes 3.45 Gbits/sec 0 943 KBytes
[ 5] 6.03-7.06 sec 425 MBytes 3.45 Gbits/sec 0 943 KBytes
[ 5] 7.06-8.06 sec 411 MBytes 3.46 Gbits/sec 0 943 KBytes
[ 5] 8.06-9.06 sec 412 MBytes 3.45 Gbits/sec 0 943 KBytes
[ 5] 9.06-10.06 sec 410 MBytes 3.44 Gbits/sec 0 1019 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.06 sec 4.04 GBytes 3.45 Gbits/sec 67 sender
[ 5] 0.00-10.06 sec 4.04 GBytes 3.45 Gbits/sec receiver
iperf Done.
iperf3 to same server as above but booted Ubuntu 22.04 (still same hardware)# iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 26181 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.06 sec 916 MBytes 7.22 Gbits/sec 58 341 KBytes
[ 5] 1.06-2.06 sec 686 MBytes 5.78 Gbits/sec 55 98.2 KBytes
[ 5] 2.06-3.06 sec 798 MBytes 6.69 Gbits/sec 62 486 KBytes
[ 5] 3.06-4.00 sec 806 MBytes 7.18 Gbits/sec 88 386 KBytes
[ 5] 4.00-5.06 sec 1.09 GBytes 8.83 Gbits/sec 102 285 KBytes
[ 5] 5.06-6.00 sec 991 MBytes 8.86 Gbits/sec 63 343 KBytes
[ 5] 6.00-7.06 sec 1.02 GBytes 8.26 Gbits/sec 20 9.91 KBytes
[ 5] 7.06-8.06 sec 977 MBytes 8.23 Gbits/sec 22 486 KBytes
[ 5] 8.06-9.06 sec 895 MBytes 7.51 Gbits/sec 165 193 KBytes
[ 5] 9.06-10.00 sec 922 MBytes 8.22 Gbits/sec 75 334 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 8.94 GBytes 7.68 Gbits/sec 710 sender
[ 5] 0.00-10.08 sec 8.94 GBytes 7.62 Gbits/sec receiver
iperf Done.
Clean FreeBSD 14.1 iperf3 to Ubuntu with 10Gbps NIC:# iperf3 -c 192.168.1.3
Connecting to host 192.168.1.3, port 5201
[ 5] local 192.168.1.1 port 24543 connected to 192.168.1.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.04 sec 1.04 GBytes 8.55 Gbits/sec 38 367 KBytes
[ 5] 1.04-2.04 sec 1.01 GBytes 8.64 Gbits/sec 26 691 KBytes
[ 5] 2.04-3.06 sec 1021 MBytes 8.42 Gbits/sec 60 694 KBytes
[ 5] 3.06-4.01 sec 963 MBytes 8.47 Gbits/sec 49 726 KBytes
[ 5] 4.01-5.00 sec 984 MBytes 8.36 Gbits/sec 54 284 KBytes
[ 5] 5.00-6.00 sec 1.05 GBytes 9.04 Gbits/sec 28 817 KBytes
[ 5] 6.00-7.06 sec 1.11 GBytes 9.01 Gbits/sec 26 394 KBytes
[ 5] 7.06-8.06 sec 1.01 GBytes 8.71 Gbits/sec 45 322 KBytes
[ 5] 8.06-9.06 sec 1015 MBytes 8.51 Gbits/sec 42 478 KBytes
[ 5] 9.06-10.06 sec 1.04 GBytes 8.93 Gbits/sec 23 123 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.06 sec 10.1 GBytes 8.67 Gbits/sec 391 sender
[ 5] 0.00-10.14 sec 10.1 GBytes 8.60 Gbits/sec receiver
iperf Done.iperf3 to same server as above but booted from fresh install of Clean FreeBSD 14.1 (still same hardware)# uname -a
FreeBSD admins-opn-2 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64
# iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 22759 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.72 GBytes 23.4 Gbits/sec 0 1.60 MBytes
[ 5] 1.00-2.00 sec 2.74 GBytes 23.5 Gbits/sec 0 1.60 MBytes
[ 5] 2.00-3.00 sec 2.74 GBytes 23.5 Gbits/sec 0 1.60 MBytes
[ 5] 3.00-4.00 sec 2.74 GBytes 23.5 Gbits/sec 0 1.60 MBytes
[ 5] 4.00-5.06 sec 2.91 GBytes 23.5 Gbits/sec 0 1.60 MBytes
[ 5] 5.06-6.06 sec 2.72 GBytes 23.5 Gbits/sec 0 1.60 MBytes
[ 5] 6.06-7.00 sec 2.59 GBytes 23.5 Gbits/sec 0 1.60 MBytes
[ 5] 7.00-8.00 sec 2.74 GBytes 23.5 Gbits/sec 0 1.61 MBytes
[ 5] 8.00-9.04 sec 2.86 GBytes 23.5 Gbits/sec 0 1.61 MBytes
[ 5] 9.04-10.06 sec 2.79 GBytes 23.5 Gbits/sec 0 1.61 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.06 sec 27.5 GBytes 23.5 Gbits/sec 0 sender
[ 5] 0.00-10.06 sec 27.5 GBytes 23.5 Gbits/sec receiver
iperf Done.
So:
- Ubuntu to Ubuntu - performance is ok about 23.5Gbps on 25Gbps NIC on both side.
- OPNsense to OPNsense - performance is terrible about 3.5Gbps on 25Gbps NIC on both side.
- OPNsense to Ubuntu - performance is about 7.5Gbps-8.5Gbps on 25Gbps NIC on both side.
- OPNsense to Ubuntu(10Gbps) - performance is about 7.5Gbps-8.5Gbps on 25Gbps NIC on OPNsense side.
- Clean FreeBSD 14.1 to OPNsense - performance is ok about 3.5Gbps on 25Gbps NIC on both side.
- Clean FreeBSD 14.1 to Ubuntu - performance is about 7.5Gbps-8.5Gbps on 25Gbps NIC on both side.
- Clean FreeBSD 14.1 to Ubuntu(10Gbps) - performance is about 7.5Gbps-8.5Gbps on 25Gbps NIC on OPNsense side.
- Clean FreeBSD 14.1 to Clean FreeBSD 14.1 - performance is ok 23.5Gbps on 25Gbps NIC on both side. Same FW 8.5 different drivers (Intels 1.14.2 on first one and FreeBSD 2.3.3-k on second)
I can do more tests if You want.
Regards
Borys
Hardware acceleration settings maybe? They default to on in FreeBSD... In general difficult to tell, but likely a small setting somewhere.
Cheers,
Franco
Quote from: franco on December 05, 2024, 07:52:38 PM
Hardware acceleration settings maybe? They default to on in FreeBSD... In general difficult to tell, but likely a small setting somewhere.
Cheers,
Franco
I don't know what should I looking for, so I made diff between sysctl -a from Clear FreeBSD (sysctl_1) and OPNsense (sysctl_2) and I don't see any spectacular differences (files in attachments).
Here is the link to a video (4 minutes) from this diff:
https://owncloud.ohnsorge.pl/index.php/s/Navn1l86y397umv/download (https://owncloud.ohnsorge.pl/index.php/s/Navn1l86y397umv/download)
Sorry, I don't expect anyone will be looking at this.
Instead take a look here:
https://docs.opnsense.org/manual/interfaces_settings.html
Cheers,
Franco
Can you try 8 parallel streams? Quite sure one stream cant handle full bandwith with nic queue handling
Quote from: franco on December 06, 2024, 09:33:33 AM
Sorry, I don't expect anyone will be looking at this.
Instead take a look here:
https://docs.opnsense.org/manual/interfaces_settings.html
Cheers,
Franco
Can You advise (if You know of course) if the basic Business Support Package (No subscription) is sufficient to report and resolve such an issue or I have to solve this problem on my own? I have access to Broadcom cards and Intel cards and I think the solution to this problem will be good and valuable to me, to this product and to the rest of the OPNsense community.
Quote from: mimugmail on December 06, 2024, 09:43:20 AM
Can you try 8 parallel streams?
Yes sure:
command: iperf3 -c 192.168.1.2 -P 8
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.02 GBytes 880 Mbits/sec 125 sender
[ 5] 0.00-10.00 sec 1.02 GBytes 879 Mbits/sec receiver
[ 7] 0.00-10.00 sec 2.29 GBytes 1.97 Gbits/sec 0 sender
[ 7] 0.00-10.00 sec 2.29 GBytes 1.97 Gbits/sec receiver
[ 9] 0.00-10.00 sec 1.03 GBytes 885 Mbits/sec 123 sender
[ 9] 0.00-10.00 sec 1.03 GBytes 884 Mbits/sec receiver
[ 11] 0.00-10.00 sec 1.28 GBytes 1.10 Gbits/sec 16 sender
[ 11] 0.00-10.00 sec 1.28 GBytes 1.10 Gbits/sec receiver
[ 13] 0.00-10.00 sec 2.95 GBytes 2.53 Gbits/sec 1 sender
[ 13] 0.00-10.00 sec 2.94 GBytes 2.53 Gbits/sec receiver
[ 15] 0.00-10.00 sec 2.28 GBytes 1.96 Gbits/sec 0 sender
[ 15] 0.00-10.00 sec 2.28 GBytes 1.96 Gbits/sec receiver
[ 17] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec 30 sender
[ 17] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec receiver
[ 19] 0.00-10.00 sec 1006 MBytes 844 Mbits/sec 106 sender
[ 19] 0.00-10.00 sec 1005 MBytes 843 Mbits/sec receiver
[SUM] 0.00-10.00 sec 13.2 GBytes 11.4 Gbits/sec 401 sender
[SUM] 0.00-10.00 sec 13.2 GBytes 11.3 Gbits/sec receiver
iperf Done.
screen of top command in attachement.
Quote from: mimugmail on December 06, 2024, 09:43:20 AM
Quite sure one stream cant handle full bandwith with nic queue handling
Ubuntu 22.04 to Ubuntu 22.04 - can
Clear FreeBSD to Clear FreeBSD - also can
Please look at my earlier tests.
By comparing both files, it looks to me as if they are using different drivers for the ixl0 interface : <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2> // ixl0: <Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k>
This generating different configuration of the network card.
Regards
Quote from: FraLem on December 07, 2024, 09:35:26 AM
By comparing both files, it looks to me as if they are using different drivers for the ixl0 interface : <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2> // ixl0: <Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k>
This generating different configuration of the network card.
Regards
I have tested on both drivers and it doesn't change anything.
When FreeBSD 14.1 to FreeBSD 14.1 is fester than OPN to OPN you should do a sysctl -a > sys.out and diff around. OPN will set defaults untouched since FreeBSD 11 (some of them). Just a guess ...
Quote from: mimugmail on December 08, 2024, 02:22:57 PM
When FreeBSD 14.1 to FreeBSD 14.1 is fester than OPN to OPN you should do a sysctl -a > sys.out and diff around. OPN will set defaults untouched since FreeBSD 11 (some of them). Just a guess ...
This comparison is on 4 min film that I previously attached link to. I don't see any spectacular differences. But I don't know what I should be paying attention to.
Sorry, late to the party, didn't read every thread. Ok, I compared both.
1st test, set on OPN (only for testing):
net.inet.tcp.delayed_ack: 0 -> 1
net.inet.tcp.blackhole: 2 -> 0
net.inet.udp.blackhole: 1 -> 0
(net.inet.rss.enabled: 0 -> 1) enable in second test
net.isr.bindthreads: 1 -> 0
hw.ibrs_disable: 0 -> 1
If this doesn't help:
Firewall : Settings : Advanced -> Disable shared forwarding, reboot, test again
If this also doesn't help:
Screenshot of hw offloading
Also observing that F14 loads newer driver where it has 4 queues compared to 8 in OPN. Just as an info.
Quote from: mimugmail on December 08, 2024, 05:40:41 PM
Sorry, late to the party, didn't read every thread. Ok, I compared both.
1st test, set on OPN (only for testing):
net.inet.tcp.delayed_ack: 0 -> 1
net.inet.tcp.blackhole: 2 -> 0
net.inet.udp.blackhole: 1 -> 0
(net.inet.rss.enabled: 0 -> 1) enable in second test
net.isr.bindthreads: 1 -> 0
hw.ibrs_disable: 0 -> 1
If this doesn't help:
Firewall : Settings : Advanced -> Disable shared forwarding, reboot, test again
If this also doesn't help:
Screenshot of hw offloading
Also observing that F14 loads newer driver where it has 4 queues compared to 8 in OPN. Just as an info.
OK, I will test it tomorrow morning.
Quote from: mimugmail on December 08, 2024, 05:40:41 PM
Sorry, late to the party, didn't read every thread. Ok, I compared both.
1st test, set on OPN (only for testing):
net.inet.tcp.delayed_ack: 0 -> 1
net.inet.tcp.blackhole: 2 -> 0
net.inet.udp.blackhole: 1 -> 0
(net.inet.rss.enabled: 0 -> 1) enable in second test
net.isr.bindthreads: 1 -> 0
hw.ibrs_disable: 0 -> 1
If this doesn't help:
Firewall : Settings : Advanced -> Disable shared forwarding, reboot, test again
If this also doesn't help:
Screenshot of hw offloading
Also observing that F14 loads newer driver where it has 4 queues compared to 8 in OPN. Just as an info.
After making the following changes it's a bit better, but it's still far from being on a pure FreeBSD.
root@admins-opn-1:~ # sysctl net.inet.tcp.delayed_ack
net.inet.tcp.delayed_ack: 1
root@admins-opn-1:~ # sysctl net.inet.tcp.blackhole
net.inet.tcp.blackhole: 0
root@admins-opn-1:~ # sysctl net.inet.udp.blackhole
net.inet.udp.blackhole: 0
root@admins-opn-1:~ # sysctl net.inet.rss.enabled
net.inet.rss.enabled: 0
root@admins-opn-1:~ # sysctl net.isr.bindthreads
net.isr.bindthreads: 0
root@admins-opn-1:~ # sysctl hw.ibrs_disable
hw.ibrs_disable: 1
iper3:
perf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 54166 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.02 sec 618 MBytes 5.10 Gbits/sec 0 1.32 MBytes
[ 5] 1.02-2.00 sec 659 MBytes 5.61 Gbits/sec 0 1.32 MBytes
[ 5] 2.00-3.00 sec 649 MBytes 5.45 Gbits/sec 0 1.32 MBytes
[ 5] 3.00-4.01 sec 654 MBytes 5.45 Gbits/sec 0 1.32 MBytes
[ 5] 4.01-5.06 sec 684 MBytes 5.44 Gbits/sec 0 1.32 MBytes
[ 5] 5.06-6.03 sec 628 MBytes 5.45 Gbits/sec 17 1021 KBytes
[ 5] 6.03-7.06 sec 672 MBytes 5.46 Gbits/sec 0 1.18 MBytes
[ 5] 7.06-8.03 sec 628 MBytes 5.43 Gbits/sec 0 1.18 MBytes
[ 5] 8.03-9.00 sec 640 MBytes 5.53 Gbits/sec 0 1.21 MBytes
[ 5] 9.00-10.02 sec 680 MBytes 5.58 Gbits/sec 0 1.28 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.02 sec 6.36 GBytes 5.45 Gbits/sec 17 sender
[ 5] 0.00-10.02 sec 6.36 GBytes 5.45 Gbits/sec receiveriperf3 with -P 8:
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.62 GBytes 2.25 Gbits/sec 73 sender
[ 5] 0.00-10.00 sec 2.62 GBytes 2.25 Gbits/sec receiver
[ 7] 0.00-10.00 sec 2.83 GBytes 2.43 Gbits/sec 116 sender
[ 7] 0.00-10.00 sec 2.83 GBytes 2.43 Gbits/sec receiver
[ 9] 0.00-10.00 sec 4.06 GBytes 3.49 Gbits/sec 17 sender
[ 9] 0.00-10.00 sec 4.06 GBytes 3.49 Gbits/sec receiver
[ 11] 0.00-10.00 sec 503 MBytes 422 Mbits/sec 248 sender
[ 11] 0.00-10.00 sec 501 MBytes 420 Mbits/sec receiver
[ 13] 0.00-10.00 sec 1.42 GBytes 1.22 Gbits/sec 302 sender
[ 13] 0.00-10.00 sec 1.42 GBytes 1.22 Gbits/sec receiver
[ 15] 0.00-10.00 sec 2.64 GBytes 2.27 Gbits/sec 46 sender
[ 15] 0.00-10.00 sec 2.64 GBytes 2.27 Gbits/sec receiver
[ 17] 0.00-10.00 sec 2.53 GBytes 2.18 Gbits/sec 10 sender
[ 17] 0.00-10.00 sec 2.53 GBytes 2.17 Gbits/sec receiver
[ 19] 0.00-10.00 sec 2.19 GBytes 1.88 Gbits/sec 357 sender
[ 19] 0.00-10.00 sec 2.19 GBytes 1.88 Gbits/sec receiver
[SUM] 0.00-10.00 sec 18.8 GBytes 16.1 Gbits/sec 1169 sender
[SUM] 0.00-10.00 sec 18.8 GBytes 16.1 Gbits/sec receiver
iperf Done.speedtest:
root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 4.134 ms
Testing download speed................................................................................
Download: 7206.26 Mbit/s
Testing upload speed...
Upload: 6746.15 Mbit/sI will add the test results after making changes that require a reboot in a moment.
After setting:
root@admins-opn-1:~ # sysctl net.inet.tcp.delayed_ack=1
net.inet.tcp.delayed_ack: 0 -> 1
root@admins-opn-1:~ # sysctl net.inet.tcp.blackhole=0
net.inet.tcp.blackhole: 2 -> 0
root@admins-opn-1:~ # sysctl net.inet.udp.blackhole=0
net.inet.udp.blackhole: 1 -> 0
root@admins-opn-1:~ # sysctl net.inet.rss.enabled
net.inet.rss.enabled: 1
root@admins-opn-1:~ # sysctl net.isr.bindthreads
net.isr.bindthreads: 0
root@admins-opn-1:~ # sysctl hw.ibrs_disable=1
hw.ibrs_disable: 0 -> 1
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 5473 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 658 MBytes 5.50 Gbits/sec 0 1.16 MBytes
[ 5] 1.00-2.05 sec 717 MBytes 5.72 Gbits/sec 0 1.16 MBytes
[ 5] 2.05-3.05 sec 674 MBytes 5.67 Gbits/sec 0 1.16 MBytes
[ 5] 3.05-4.00 sec 638 MBytes 5.64 Gbits/sec 0 1.16 MBytes
[ 5] 4.00-5.05 sec 750 MBytes 5.99 Gbits/sec 0 1.16 MBytes
[ 5] 5.05-6.06 sec 689 MBytes 5.73 Gbits/sec 0 1.16 MBytes
[ 5] 6.06-7.00 sec 638 MBytes 5.70 Gbits/sec 0 1.16 MBytes
[ 5] 7.00-8.04 sec 704 MBytes 5.67 Gbits/sec 0 1.16 MBytes
[ 5] 8.04-9.00 sec 649 MBytes 5.68 Gbits/sec 0 1.16 MBytes
[ 5] 9.00-10.00 sec 687 MBytes 5.75 Gbits/sec 0 1.16 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.65 GBytes 5.71 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 6.65 GBytes 5.71 Gbits/sec receiver
iperf Done.
[SUM] 0.00-10.00 sec 18.8 GBytes 16.1 Gbits/sec 1169 sender
[SUM] 0.00-10.00 sec 18.8 GBytes 16.1 Gbits/sec receiver
We're getting closer ... slowly :)
Quote from: mimugmail on December 09, 2024, 03:41:10 PM
[SUM] 0.00-10.00 sec 18.8 GBytes 16.1 Gbits/sec 1169 sender
[SUM] 0.00-10.00 sec 18.8 GBytes 16.1 Gbits/sec receiver
We're getting closer ... slowly :)
Very slowly, but I am still very grateful for your time, willingness and attempt to help.
OPNsense: (iperf3 -c x.x.x.x)
(https://owncloud.ohnsorge.pl/index.php/s/BHc0y75Mg4lxpoK/download)
FreeBSD: (iperf3 -c x.x.x.x)
(https://owncloud.ohnsorge.pl/index.php/s/QnPloshqaWhoDh6/download)
FreeBSD: (iperf3 -c x.x.x.x -P 8 )
(https://owncloud.ohnsorge.pl/index.php/s/dXmwl6WVna114be/download)
FreeBSD: (iperf3 -c x.x.x.x -P 8 )
(https://owncloud.ohnsorge.pl/index.php/s/vcbkyUbhRRnQYE8/download)
OPN -P8 also?
Did you try disable shared forwarding?
Quote from: mimugmail on December 09, 2024, 06:02:55 PM
OPN -P8 also?
Yep(https://owncloud.ohnsorge.pl/index.php/s/KPMtTv5pbi2rO91/download)
(https://owncloud.ohnsorge.pl/index.php/s/8JvqZWymgzl9Q82/download)
(https://owncloud.ohnsorge.pl/index.php/s/1rKHlWPIaIC7K21/download)
Quote from: mimugmail on December 09, 2024, 06:02:55 PM
Did you try disable shared forwarding?
After disabling shared forwarding:(https://owncloud.ohnsorge.pl/index.php/s/roUZDFUse5WXxDH/download)
Interesting thing, below are the results from a freshly installed pfSense (no tuning/modifications):
[2.7.2-RELEASE][admin@pfSense1.home.local]/root: iperf3 -c x.x.x.2
Connecting to host x.x.x.2, port 5201
[ 5] local x.x.x.1 port 63784 connected to x.x.x.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 640 MBytes 5.36 Gbits/sec 32 631 KBytes
[ 5] 1.00-2.00 sec 680 MBytes 5.71 Gbits/sec 0 653 KBytes
[ 5] 2.00-3.00 sec 678 MBytes 5.69 Gbits/sec 0 653 KBytes
[ 5] 3.00-4.00 sec 677 MBytes 5.67 Gbits/sec 1 559 KBytes
[ 5] 4.00-5.00 sec 676 MBytes 5.68 Gbits/sec 0 559 KBytes
[ 5] 5.00-6.00 sec 676 MBytes 5.68 Gbits/sec 0 559 KBytes
[ 5] 6.00-7.00 sec 674 MBytes 5.66 Gbits/sec 0 636 KBytes
[ 5] 7.00-8.00 sec 677 MBytes 5.67 Gbits/sec 1 547 KBytes
[ 5] 8.00-9.00 sec 682 MBytes 5.73 Gbits/sec 0 551 KBytes
[ 5] 9.00-10.00 sec 682 MBytes 5.72 Gbits/sec 1 425 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.58 GBytes 5.66 Gbits/sec 35 sender
[ 5] 0.00-10.00 sec 6.58 GBytes 5.65 Gbits/sec receiver
iperf Done.
[2.7.2-RELEASE][admin@pfSense1.home.local]/root: pfctl -d
pf disabled
[2.7.2-RELEASE][admin@pfSense1.home.local]/root: iperf3 -c x.x.x.2
Connecting to host x.x.x.2, port 5201
[ 5] local x.x.x.1 port 22755 connected to x.x.x.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 797 MBytes 6.67 Gbits/sec 0 619 KBytes
[ 5] 1.00-2.00 sec 806 MBytes 6.76 Gbits/sec 0 619 KBytes
[ 5] 2.00-3.00 sec 796 MBytes 6.68 Gbits/sec 1 454 KBytes
[ 5] 3.00-4.00 sec 806 MBytes 6.76 Gbits/sec 0 454 KBytes
[ 5] 4.00-5.00 sec 809 MBytes 6.79 Gbits/sec 0 454 KBytes
[ 5] 5.00-6.00 sec 807 MBytes 6.77 Gbits/sec 0 454 KBytes
[ 5] 6.00-7.00 sec 805 MBytes 6.76 Gbits/sec 0 454 KBytes
[ 5] 7.00-8.00 sec 809 MBytes 6.78 Gbits/sec 0 454 KBytes
[ 5] 8.00-9.00 sec 802 MBytes 6.74 Gbits/sec 1 484 KBytes
[ 5] 9.00-10.00 sec 813 MBytes 6.81 Gbits/sec 0 533 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 7.86 GBytes 6.75 Gbits/sec 2 sender
[ 5] 0.00-10.00 sec 7.86 GBytes 6.75 Gbits/sec receiver
iperf Done.
[2.7.2-RELEASE][admin@pfSense1.home.local]/root: pfctl -e
pf enabled
[2.7.2-RELEASE][admin@pfSense1.home.local]/root: iperf3 -c x.x.x.2 -P 8
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec receiver
[ 7] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec 0 sender
[ 7] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec receiver
[ 9] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec 0 sender
[ 9] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec receiver
[ 11] 0.00-10.01 sec 1.09 GBytes 938 Mbits/sec 0 sender
[ 11] 0.00-10.01 sec 1.09 GBytes 938 Mbits/sec receiver
[ 13] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec 0 sender
[ 13] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec receiver
[ 15] 0.00-10.01 sec 1.09 GBytes 936 Mbits/sec 0 sender
[ 15] 0.00-10.01 sec 1.09 GBytes 936 Mbits/sec receiver
[ 17] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec 0 sender
[ 17] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec receiver
[ 19] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec 0 sender
[ 19] 0.00-10.01 sec 1.09 GBytes 937 Mbits/sec receiver
[SUM] 0.00-10.01 sec 8.73 GBytes 7.50 Gbits/sec 0 sender
[SUM] 0.00-10.01 sec 8.73 GBytes 7.50 Gbits/sec receiver
iperf Done.
[2.7.2-RELEASE][admin@pfSense1.home.local]/root: pfctl -d
pf disabled
[2.7.2-RELEASE][admin@pfSense1.home.local]/root: iperf3 -c x.x.x.2 -P 8
Connecting to host x.x.x.2, port 5201
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec 13 sender
[ 5] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec receiver
[ 7] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec 0 sender
[ 7] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec receiver
[ 9] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec 0 sender
[ 9] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec receiver
[ 11] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec 13 sender
[ 11] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec receiver
[ 13] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec 0 sender
[ 13] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec receiver
[ 15] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec 1 sender
[ 15] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec receiver
[ 17] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec 0 sender
[ 17] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec receiver
[ 19] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec 0 sender
[ 19] 0.00-10.00 sec 1.38 GBytes 1.19 Gbits/sec receiver
[SUM] 0.00-10.00 sec 11.1 GBytes 9.50 Gbits/sec 27 sender
[SUM] 0.00-10.00 sec 11.1 GBytes 9.50 Gbits/sec receiver
iperf Done.
What the ..., I know that it's still a long way to 25Gbps, which is what I am aiming for, but still, it's almost 2x better than on "pure" OPNsense.
There is a slight chance I get a test system with a bmxt 25G connected to a Mellanox Switch, will take a bit.
Problem is that iperf directly on OPN is terribly slow compared to traffic just passing through
Small success
root@admins-opn-1:~ # iperf3 -c x.x.x.2
Connecting to host x.x.x.2, port 5201
[ 5] local x.x.x.1 port 35935 connected to x.x.x.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.43 GBytes 12.3 Gbits/sec 0 2.00 MBytes
[ 5] 1.00-2.00 sec 1.44 GBytes 12.4 Gbits/sec 0 2.00 MBytes
[ 5] 2.00-3.06 sec 1.54 GBytes 12.4 Gbits/sec 0 2.00 MBytes
[ 5] 3.06-4.06 sec 1.43 GBytes 12.4 Gbits/sec 0 2.00 MBytes
[ 5] 4.06-5.06 sec 1.43 GBytes 12.4 Gbits/sec 0 2.00 MBytes
[ 5] 5.06-6.00 sec 1.36 GBytes 12.3 Gbits/sec 0 2.00 MBytes
[ 5] 6.00-7.03 sec 1.47 GBytes 12.3 Gbits/sec 0 2.00 MBytes
[ 5] 7.03-8.06 sec 1.48 GBytes 12.3 Gbits/sec 0 2.00 MBytes
[ 5] 8.06-9.02 sec 1.37 GBytes 12.4 Gbits/sec 0 2.00 MBytes
[ 5] 9.02-10.04 sec 1.47 GBytes 12.4 Gbits/sec 0 2.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 14.4 GBytes 12.3 Gbits/sec 0 sender
[ 5] 0.00-10.04 sec 14.4 GBytes 12.3 Gbits/sec receiver
iperf Done.
root@admins-opn-1:~ # iperf3 -c x.x.x.2 -P8
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.86 GBytes 1.59 Gbits/sec 277 sender
[ 5] 0.00-10.00 sec 1.85 GBytes 1.59 Gbits/sec receiver
[ 7] 0.00-10.00 sec 6.50 GBytes 5.58 Gbits/sec 2 sender
[ 7] 0.00-10.00 sec 6.50 GBytes 5.58 Gbits/sec receiver
[ 9] 0.00-10.00 sec 730 MBytes 612 Mbits/sec 112 sender
[ 9] 0.00-10.00 sec 728 MBytes 611 Mbits/sec receiver
[ 11] 0.00-10.00 sec 2.47 GBytes 2.12 Gbits/sec 247 sender
[ 11] 0.00-10.00 sec 2.47 GBytes 2.12 Gbits/sec receiver
[ 13] 0.00-10.00 sec 2.61 GBytes 2.24 Gbits/sec 267 sender
[ 13] 0.00-10.00 sec 2.61 GBytes 2.24 Gbits/sec receiver
[ 15] 0.00-10.00 sec 2.21 GBytes 1.90 Gbits/sec 270 sender
[ 15] 0.00-10.00 sec 2.21 GBytes 1.90 Gbits/sec receiver
[ 17] 0.00-10.00 sec 5.82 GBytes 5.00 Gbits/sec 0 sender
[ 17] 0.00-10.00 sec 5.82 GBytes 4.99 Gbits/sec receiver
[ 19] 0.00-10.00 sec 2.18 GBytes 1.87 Gbits/sec 270 sender
[ 19] 0.00-10.00 sec 2.18 GBytes 1.87 Gbits/sec receiver
[SUM] 0.00-10.00 sec 24.4 GBytes 20.9 Gbits/sec 1445 sender
[SUM] 0.00-10.00 sec 24.3 GBytes 20.9 Gbits/sec receiver
iperf Done.
And what did you change?
Quote from: mimugmail on December 14, 2024, 07:27:58 AMAnd what did you change?
I'm ashamed to admit it, but it seems that the main problem was the MTU settings. Currently, I achieved these results on a completely clean installation (couple of firewall rules).
On the switch side, I had the MTU set "fixed" to 9214 by my network team and I set the same on the OPNs side. Unfortunately, the effect of this was as you saw earlier. After setting the MTU to 8800 on the OPNsense side, everything started working except for pfSync (on unicast). This connection started working when I set the MTU to 1500 on the interface dedicated to HA and synchronization.
I still want to do some fine tuning and see if I can achieve even better results. When I have some free time, I will try to test the same settings for Broadcom cards as well.
So called jumbo frames were a bad idea to begin with.
More tuning&testing
hw.ibrs_disable=1
kern.ipc.maxsockbuf=614400000
net.isr.maxthreads=-1
net.isr.bindthreads=1
net.inet.rss.enabled=1
net.inet.tcp.soreceive_stream=1
net.inet.tcp.delayed_ack=1
net.inet.tcp.blackhole=0
net.inet.udp.blackhole=0
root@admins-opn-1:~ # iperf3 -c x.x.x.2
Connecting to host x.x.x.2, port 5201
[ 5] local x.x.x.1 port 20279 connected to x.x.x.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.20 GBytes 18.9 Gbits/sec 0 2.01 MBytes
[ 5] 1.00-2.03 sec 2.30 GBytes 19.2 Gbits/sec 0 2.01 MBytes
[ 5] 2.03-3.03 sec 2.26 GBytes 19.3 Gbits/sec 0 2.01 MBytes
[ 5] 3.03-4.01 sec 2.22 GBytes 19.5 Gbits/sec 0 2.01 MBytes
[ 5] 4.01-5.00 sec 2.22 GBytes 19.2 Gbits/sec 0 2.01 MBytes
[ 5] 5.00-6.00 sec 2.22 GBytes 19.1 Gbits/sec 0 2.01 MBytes
[ 5] 6.00-7.00 sec 2.24 GBytes 19.2 Gbits/sec 0 2.01 MBytes
[ 5] 7.00-8.00 sec 2.24 GBytes 19.3 Gbits/sec 0 2.01 MBytes
[ 5] 8.00-9.01 sec 2.27 GBytes 19.4 Gbits/sec 0 2.01 MBytes
[ 5] 9.01-10.00 sec 2.25 GBytes 19.5 Gbits/sec 0 2.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 22.4 GBytes 19.3 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 22.4 GBytes 19.3 Gbits/sec receiver
iperf Done.
root@admins-opn-1:~ # iperf3 -c x.x.x.2 -P4
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 7.24 GBytes 6.22 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 7.24 GBytes 6.21 Gbits/sec receiver
[ 7] 0.00-10.00 sec 6.68 GBytes 5.74 Gbits/sec 0 sender
[ 7] 0.00-10.00 sec 6.68 GBytes 5.73 Gbits/sec receiver
[ 9] 0.00-10.00 sec 7.24 GBytes 6.22 Gbits/sec 0 sender
[ 9] 0.00-10.00 sec 7.24 GBytes 6.22 Gbits/sec receiver
[ 11] 0.00-10.00 sec 7.24 GBytes 6.22 Gbits/sec 0 sender
[ 11] 0.00-10.00 sec 7.24 GBytes 6.21 Gbits/sec receiver
[SUM] 0.00-10.00 sec 28.4 GBytes 24.4 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 28.4 GBytes 24.4 Gbits/sec receiver
root@admins-opn-1:~ # iperf3 -c x.x.x.2 -P8
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.82 GBytes 1.56 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.81 GBytes 1.56 Gbits/sec receiver
[ 7] 0.00-10.00 sec 1.81 GBytes 1.55 Gbits/sec 0 sender
[ 7] 0.00-10.00 sec 1.81 GBytes 1.55 Gbits/sec receiver
[ 9] 0.00-10.00 sec 3.61 GBytes 3.10 Gbits/sec 0 sender
[ 9] 0.00-10.00 sec 3.61 GBytes 3.10 Gbits/sec receiver
[ 11] 0.00-10.00 sec 7.24 GBytes 6.22 Gbits/sec 0 sender
[ 11] 0.00-10.00 sec 7.24 GBytes 6.21 Gbits/sec receiver
[ 13] 0.00-10.00 sec 6.82 GBytes 5.85 Gbits/sec 0 sender
[ 13] 0.00-10.00 sec 6.82 GBytes 5.85 Gbits/sec receiver
[ 15] 0.00-10.00 sec 1.82 GBytes 1.56 Gbits/sec 0 sender
[ 15] 0.00-10.00 sec 1.81 GBytes 1.56 Gbits/sec receiver
[ 17] 0.00-10.00 sec 3.63 GBytes 3.12 Gbits/sec 0 sender
[ 17] 0.00-10.00 sec 3.63 GBytes 3.12 Gbits/sec receiver
[ 19] 0.00-10.00 sec 1.81 GBytes 1.56 Gbits/sec 0 sender
[ 19] 0.00-10.00 sec 1.81 GBytes 1.56 Gbits/sec receiver
[SUM] 0.00-10.00 sec 28.6 GBytes 24.5 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 28.5 GBytes 24.5 Gbits/sec receiver
iperf Done.
Looks good.
Would you share final interfaces configuration on Opnsense (ifconfig) as well as Driver used on the NIcs?
Thanks & regards
Quote from: FraLem on December 22, 2024, 09:03:11 AMLooks good.
Would you share final interfaces configuration on Opnsense (ifconfig) as well as Driver used on the NIcs?
Thanks & regards
Yes Sure.
root@admins-opn-1:~ # ifconfig
ixl0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether xx:xx:xx:xx:11:b4
media: Ethernet autoselect (25GBase-LR <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
ixl1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether xx:xx:xx:xx:11:b4
hwaddr xx:xx:xx:xx:11:b5
media: Ethernet autoselect (25GBase-CR <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
igb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
ether xx:xx:xx:xx:22:24
media: Ethernet autoselect
status: no carrier
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
ether xx:xx:xx:xx:22:25
media: Ethernet autoselect
status: no carrier
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0 metric 0 mtu 1536
options=0
groups: enc
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
pfsync0: flags=1000041<UP,RUNNING,LOWER_UP> metric 0 mtu 1500
options=0
syncdev: lagg0_vlan2 syncpeer: 192.168.2.249 maxupd: 128 defer: off version: 1400
syncok: 1
groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33152
options=0
groups: pflog
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether xx:xx:xx:xx:11:b4
hwaddr 00:00:00:00:00:00
inet6 xx:xx:xx:xx%lagg0 prefixlen 64 scopeid 0x9
laggproto lacp lagghash l2,l3
laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lagg0_vlanXX: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: lagg0_vlanXX (opt2)
options=4000000<MEXTPG>
ether xx:xx:xx:xx:11:b4
inet x.x.x.x netmask 0xfffffff0 broadcast x.x.x.x
inet6 xx:xx:xx:xx%lagg0_vlanXX prefixlen 64 scopeid 0xa
groups: vlan
vlan: XX vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: lagg0_vlan2 (opt1)
options=4000000<MEXTPG>
ether xx:xx:xx:xx:11:b4
inet 192.168.2.241 netmask 0xffffff00 broadcast 192.168.2.255
inet6 xx:xx:xx:xx%lagg0_vlan2 prefixlen 64 scopeid 0xb
groups: vlan
vlan: 2 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan3: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
description: lagg0_vlan3 (opt6)
options=4000000<MEXTPG>
ether xx:xx:xx:xx:11:b4
inet 10.255.5.2 netmask 0xffffff00 broadcast 10.255.5.255
inet 10.255.5.1 netmask 0xffffff00 broadcast 10.255.5.255 vhid 91
inet6 xx:xx:xx:xx%lagg0_vlan3 prefixlen 64 scopeid 0xc
inet6 xx.xx.xx.xx::2 prefixlen 64
inet6 xx.xx.xx.xx::1 prefixlen 64 vhid 92
groups: vlan
carp: MASTER vhid 91 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
carp: MASTER vhid 92 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
vlan: 3 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
lagg0_vlan4: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
options=4000000<MEXTPG>
ether xx:xx:xx:xx:11:b4
inet6 xx:xx:xx:xx%lagg0_vlan4 prefixlen 64 scopeid 0xd
groups: vlan
vlan: 4 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lagg0_vlan52: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
description: lagg0_vlan52 (wan)
options=4000000<MEXTPG>
ether xx:xx:xx:xx:11:b4
inet x.x.x.201 netmask 0xffffffe0 broadcast x.x.x.223
inet x.x.x.200 netmask 0xffffffe0 broadcast x.x.x.223 vhid 98
inet6 xx:xx:xx:xx%lagg0_vlan52 prefixlen 64 scopeid 0xf
inet6 xxxx:yyyy:zzzz::11 prefixlen 64
inet6 xxxx:yyyy:zzzz::10 prefixlen 64 vhid 99
groups: vlan
carp: MASTER vhid 98 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
carp: MASTER vhid 99 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
vlan: 52 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
root@admins-opn-1:~ #
FW: 9.30 default OPNsense ixl driver
root@admins-opn-1:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.ixl.1.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k
dev.ixl.0.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k