So I was doing some playing today and noticed on my bare metal firewall I am getting 3-4Gbit/s AFTER a reboot using any interface. My server hardwar information can be found at https://bsd-hardware.info/?probe=30789867a9. These interfaces are X722 10Gbe running to an Arista DCS-7050T-64-R and the interfaces are configured as follows:
interface Ethernet47
description Firewall - ixl0 - vl20,vl30,vl40,vl50
mtu 9198
switchport access vlan 20
interface Ethernet48
description Firewall - ixl1 - vl20,vl30,vl40,vl50
mtu 9198
switchport trunk allowed vlan 20,30,40,50,60
switchport mode trunk
Simple network configuration because I've been pulling my hair out trying to elimnate various variables (e.g., LAGG, VLAN, and even Straight Interfaces).
Now here is the weird part... On reboot I get the 3-4Gbit/s speeds BUT the moment I make some type of interface change the performance goes to the expected ~9Gbit/s.
These are the iperf speeds talking to devices on the same subnet so there is no intervlan traffic here so it should be nearly line speed of course but see how bad they are:
iperf3 -P8 going out ixl0, no VLANs:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 509 MBytes 427 Mbits/sec 11 sender
[ 5] 0.00-10.00 sec 509 MBytes 427 Mbits/sec receiver
[ 7] 0.00-10.00 sec 510 MBytes 428 Mbits/sec 14 sender
[ 7] 0.00-10.00 sec 510 MBytes 428 Mbits/sec receiver
[ 9] 0.00-10.00 sec 511 MBytes 428 Mbits/sec 23 sender
[ 9] 0.00-10.00 sec 511 MBytes 428 Mbits/sec receiver
[ 11] 0.00-10.00 sec 509 MBytes 427 Mbits/sec 17 sender
[ 11] 0.00-10.00 sec 509 MBytes 427 Mbits/sec receiver
[ 13] 0.00-10.00 sec 510 MBytes 428 Mbits/sec 16 sender
[ 13] 0.00-10.00 sec 510 MBytes 428 Mbits/sec receiver
[ 15] 0.00-10.00 sec 510 MBytes 428 Mbits/sec 14 sender
[ 15] 0.00-10.00 sec 510 MBytes 428 Mbits/sec receiver
[ 17] 0.00-10.00 sec 509 MBytes 427 Mbits/sec 9 sender
[ 17] 0.00-10.00 sec 509 MBytes 427 Mbits/sec receiver
[ 19] 0.00-10.00 sec 510 MBytes 428 Mbits/sec 13 sender
[ 19] 0.00-10.00 sec 510 MBytes 428 Mbits/sec receiver
[SUM] 0.00-10.00 sec 3.98 GBytes 3.42 Gbits/sec 117 sender
[SUM] 0.00-10.00 sec 3.98 GBytes 3.42 Gbits/sec receiver
iperf3 -P8 going out ixl1_vlan30:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 487 MBytes 408 Mbits/sec 4 sender
[ 5] 0.00-10.01 sec 487 MBytes 408 Mbits/sec receiver
[ 7] 0.00-10.01 sec 487 MBytes 408 Mbits/sec 9 sender
[ 7] 0.00-10.01 sec 487 MBytes 408 Mbits/sec receiver
[ 9] 0.00-10.01 sec 487 MBytes 408 Mbits/sec 5 sender
[ 9] 0.00-10.01 sec 487 MBytes 408 Mbits/sec receiver
[ 11] 0.00-10.01 sec 483 MBytes 404 Mbits/sec 8 sender
[ 11] 0.00-10.01 sec 483 MBytes 404 Mbits/sec receiver
[ 13] 0.00-10.01 sec 487 MBytes 408 Mbits/sec 0 sender
[ 13] 0.00-10.01 sec 487 MBytes 408 Mbits/sec receiver
[ 15] 0.00-10.01 sec 487 MBytes 408 Mbits/sec 0 sender
[ 15] 0.00-10.01 sec 487 MBytes 408 Mbits/sec receiver
[ 17] 0.00-10.01 sec 487 MBytes 408 Mbits/sec 0 sender
[ 17] 0.00-10.01 sec 487 MBytes 408 Mbits/sec receiver
[ 19] 0.00-10.01 sec 482 MBytes 404 Mbits/sec 0 sender
[ 19] 0.00-10.01 sec 482 MBytes 404 Mbits/sec receiver
[SUM] 0.00-10.01 sec 3.80 GBytes 3.26 Gbits/sec 26 sender
[SUM] 0.00-10.01 sec 3.80 GBytes 3.26 Gbits/sec receiver
root@firewall:~ # ifconfig ixl0
ixl0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether ac:1f:6b:c6:3c:0a
inet 192.168.50.1 netmask 0xfffffe00 broadcast 192.168.51.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@firewall:~ # ifconfig ixl1_vlan30
ixl1_vlan30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: IoT
options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether ac:1f:6b:c6:3c:0b
inet 192.168.54.1 netmask 0xfffffe00 broadcast 192.168.55.255
groups: vlan
vlan: 30 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl1
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@firewall:~ #
Now I moved the LAN interface (VLAN20) from the dedicated ixl0 to ixl1_vlan20 and the performance jumps back to normal as shown below:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.39 GBytes 1.20 Gbits/sec 31 sender
[ 5] 0.00-10.00 sec 1.39 GBytes 1.20 Gbits/sec receiver
[ 7] 0.00-10.00 sec 1.29 GBytes 1.11 Gbits/sec 26 sender
[ 7] 0.00-10.00 sec 1.29 GBytes 1.11 Gbits/sec receiver
[ 9] 0.00-10.00 sec 1.30 GBytes 1.11 Gbits/sec 34 sender
[ 9] 0.00-10.00 sec 1.29 GBytes 1.11 Gbits/sec receiver
[ 11] 0.00-10.00 sec 1.33 GBytes 1.14 Gbits/sec 36 sender
[ 11] 0.00-10.00 sec 1.33 GBytes 1.14 Gbits/sec receiver
[ 13] 0.00-10.00 sec 1.23 GBytes 1.06 Gbits/sec 26 sender
[ 13] 0.00-10.00 sec 1.23 GBytes 1.05 Gbits/sec receiver
[ 15] 0.00-10.00 sec 1.55 GBytes 1.33 Gbits/sec 20 sender
[ 15] 0.00-10.00 sec 1.55 GBytes 1.33 Gbits/sec receiver
[ 17] 0.00-10.00 sec 1.47 GBytes 1.27 Gbits/sec 56 sender
[ 17] 0.00-10.00 sec 1.47 GBytes 1.26 Gbits/sec receiver
[ 19] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec 29 sender
[ 19] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec receiver
[SUM] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 258 sender
[SUM] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec receiver
[root@firewall ~]# ifconfig ixl0
ixl0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
ether ac:1f:6b:c6:3c:0a
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
[root@firewall ~]# ifconfig ixl1_vlan20
ixl1_vlan20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether ac:1f:6b:c6:3c:0b
inet 192.168.50.1 netmask 0xfffffe00 broadcast 192.168.51.255
groups: vlan
vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl1
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
[root@firewall ~]#
But if I run the iperf on the vlan30 interface, that wasn't touched, but attached via VLANs to the SAME interface now as vlan20 you can see the speeds are in "werid" state. The ifconfig of that interface is already posted above.
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 486 MBytes 408 Mbits/sec 5 sender
[ 5] 0.00-10.01 sec 486 MBytes 408 Mbits/sec receiver
[ 7] 0.00-10.01 sec 486 MBytes 407 Mbits/sec 3 sender
[ 7] 0.00-10.01 sec 486 MBytes 407 Mbits/sec receiver
[ 9] 0.00-10.01 sec 486 MBytes 408 Mbits/sec 6 sender
[ 9] 0.00-10.01 sec 486 MBytes 408 Mbits/sec receiver
[ 11] 0.00-10.01 sec 486 MBytes 408 Mbits/sec 13 sender
[ 11] 0.00-10.01 sec 486 MBytes 408 Mbits/sec receiver
[ 13] 0.00-10.01 sec 486 MBytes 407 Mbits/sec 0 sender
[ 13] 0.00-10.01 sec 486 MBytes 407 Mbits/sec receiver
[ 15] 0.00-10.01 sec 486 MBytes 407 Mbits/sec 0 sender
[ 15] 0.00-10.01 sec 486 MBytes 407 Mbits/sec receiver
[ 17] 0.00-10.01 sec 486 MBytes 408 Mbits/sec 0 sender
[ 17] 0.00-10.01 sec 486 MBytes 408 Mbits/sec receiver
[ 19] 0.00-10.01 sec 486 MBytes 408 Mbits/sec 0 sender
[ 19] 0.00-10.01 sec 486 MBytes 408 Mbits/sec receiver
[SUM] 0.00-10.01 sec 3.80 GBytes 3.26 Gbits/sec 27 sender
[SUM] 0.00-10.01 sec 3.80 GBytes 3.26 Gbits/sec receiver
If I do something with this interface such as reassign it to another interface or even reassign the vlan to another interface the speeds will return to the expected state.
Now if I reboot ALL interfaces will be in this slower state until moved around but they will retain this fixed state on the touched interfaces until the next reboot.
Hardware CRC, Hardware TSO, and Hardware LRO are all disabled and VLAN Hardware Filtering is set to Leave default. All my system tuneables are set to default right now.
I am so confused and not sure what is going on but I need help and not sure what else to do. Willing to provide any additional debug information needed to help troubleshoot and test.
Thanks!