Weird performance issues on interfaces until change occurs...

[pyrodex]

So I was doing some playing today and noticed on my bare metal firewall I am getting 3-4Gbit/s AFTER a reboot using any interface. My server hardwar information can be found at https://bsd-hardware.info/?probe=30789867a9. These interfaces are X722 10Gbe running to an Arista DCS-7050T-64-R and the interfaces are configured as follows:

Code Select Expand


interface Ethernet47
   description Firewall - ixl0 - vl20,vl30,vl40,vl50
   mtu 9198
   switchport access vlan 20
interface Ethernet48
   description Firewall - ixl1 - vl20,vl30,vl40,vl50
   mtu 9198
   switchport trunk allowed vlan 20,30,40,50,60
   switchport mode trunk


Simple network configuration because I've been pulling my hair out trying to elimnate various variables (e.g., LAGG, VLAN, and even Straight Interfaces).

Now here is the weird part... On reboot I get the 3-4Gbit/s speeds BUT the moment I make some type of interface change the performance goes to the expected ~9Gbit/s.

These are the iperf speeds talking to devices on the same subnet so there is no intervlan traffic here so it should be nearly line speed of course but see how bad they are:

iperf3 -P8 going out ixl0, no VLANs:

Code Select Expand


[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   509 MBytes   427 Mbits/sec   11             sender
[  5]   0.00-10.00  sec   509 MBytes   427 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   510 MBytes   428 Mbits/sec   14             sender
[  7]   0.00-10.00  sec   510 MBytes   428 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   511 MBytes   428 Mbits/sec   23             sender
[  9]   0.00-10.00  sec   511 MBytes   428 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   509 MBytes   427 Mbits/sec   17             sender
[ 11]   0.00-10.00  sec   509 MBytes   427 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec   510 MBytes   428 Mbits/sec   16             sender
[ 13]   0.00-10.00  sec   510 MBytes   428 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec   510 MBytes   428 Mbits/sec   14             sender
[ 15]   0.00-10.00  sec   510 MBytes   428 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec   509 MBytes   427 Mbits/sec    9             sender
[ 17]   0.00-10.00  sec   509 MBytes   427 Mbits/sec                  receiver
[ 19]   0.00-10.00  sec   510 MBytes   428 Mbits/sec   13             sender
[ 19]   0.00-10.00  sec   510 MBytes   428 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  3.98 GBytes  3.42 Gbits/sec  117             sender
[SUM]   0.00-10.00  sec  3.98 GBytes  3.42 Gbits/sec                  receiver


iperf3 -P8 going out ixl1_vlan30:

Code Select Expand


[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   487 MBytes   408 Mbits/sec    4             sender
[  5]   0.00-10.01  sec   487 MBytes   408 Mbits/sec                  receiver
[  7]   0.00-10.01  sec   487 MBytes   408 Mbits/sec    9             sender
[  7]   0.00-10.01  sec   487 MBytes   408 Mbits/sec                  receiver
[  9]   0.00-10.01  sec   487 MBytes   408 Mbits/sec    5             sender
[  9]   0.00-10.01  sec   487 MBytes   408 Mbits/sec                  receiver
[ 11]   0.00-10.01  sec   483 MBytes   404 Mbits/sec    8             sender
[ 11]   0.00-10.01  sec   483 MBytes   404 Mbits/sec                  receiver
[ 13]   0.00-10.01  sec   487 MBytes   408 Mbits/sec    0             sender
[ 13]   0.00-10.01  sec   487 MBytes   408 Mbits/sec                  receiver
[ 15]   0.00-10.01  sec   487 MBytes   408 Mbits/sec    0             sender
[ 15]   0.00-10.01  sec   487 MBytes   408 Mbits/sec                  receiver
[ 17]   0.00-10.01  sec   487 MBytes   408 Mbits/sec    0             sender
[ 17]   0.00-10.01  sec   487 MBytes   408 Mbits/sec                  receiver
[ 19]   0.00-10.01  sec   482 MBytes   404 Mbits/sec    0             sender
[ 19]   0.00-10.01  sec   482 MBytes   404 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  3.80 GBytes  3.26 Gbits/sec   26             sender
[SUM]   0.00-10.01  sec  3.80 GBytes  3.26 Gbits/sec                  receiver


Code Select Expand


root@firewall:~ # ifconfig ixl0
ixl0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether ac:1f:6b:c6:3c:0a
inet 192.168.50.1 netmask 0xfffffe00 broadcast 192.168.51.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@firewall:~ # ifconfig ixl1_vlan30
ixl1_vlan30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: IoT
options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether ac:1f:6b:c6:3c:0b
inet 192.168.54.1 netmask 0xfffffe00 broadcast 192.168.55.255
groups: vlan
vlan: 30 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl1
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@firewall:~ #



Now I moved the LAN interface (VLAN20) from the dedicated ixl0 to ixl1_vlan20 and the performance jumps back to normal as shown below:

Code Select Expand


[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.39 GBytes  1.20 Gbits/sec   31             sender
[  5]   0.00-10.00  sec  1.39 GBytes  1.20 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  1.29 GBytes  1.11 Gbits/sec   26             sender
[  7]   0.00-10.00  sec  1.29 GBytes  1.11 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  1.30 GBytes  1.11 Gbits/sec   34             sender
[  9]   0.00-10.00  sec  1.29 GBytes  1.11 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  1.33 GBytes  1.14 Gbits/sec   36             sender
[ 11]   0.00-10.00  sec  1.33 GBytes  1.14 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  1.23 GBytes  1.06 Gbits/sec   26             sender
[ 13]   0.00-10.00  sec  1.23 GBytes  1.05 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  1.55 GBytes  1.33 Gbits/sec   20             sender
[ 15]   0.00-10.00  sec  1.55 GBytes  1.33 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  1.47 GBytes  1.27 Gbits/sec   56             sender
[ 17]   0.00-10.00  sec  1.47 GBytes  1.26 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  1.37 GBytes  1.18 Gbits/sec   29             sender
[ 19]   0.00-10.00  sec  1.37 GBytes  1.18 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec  258             sender
[SUM]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec                  receiver


Code Select Expand


[root@firewall ~]# ifconfig ixl0
ixl0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,NOMAP>
        ether ac:1f:6b:c6:3c:0a
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
[root@firewall ~]# ifconfig ixl1_vlan20
ixl1_vlan20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether ac:1f:6b:c6:3c:0b
        inet 192.168.50.1 netmask 0xfffffe00 broadcast 192.168.51.255
        groups: vlan
        vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl1
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
[root@firewall ~]#


But if I run the iperf on the vlan30 interface, that wasn't touched, but attached via VLANs to the SAME interface now as vlan20 you can see the speeds are in "werid" state. The ifconfig of that interface is already posted above.

Code Select Expand


[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   486 MBytes   408 Mbits/sec    5             sender
[  5]   0.00-10.01  sec   486 MBytes   408 Mbits/sec                  receiver
[  7]   0.00-10.01  sec   486 MBytes   407 Mbits/sec    3             sender
[  7]   0.00-10.01  sec   486 MBytes   407 Mbits/sec                  receiver
[  9]   0.00-10.01  sec   486 MBytes   408 Mbits/sec    6             sender
[  9]   0.00-10.01  sec   486 MBytes   408 Mbits/sec                  receiver
[ 11]   0.00-10.01  sec   486 MBytes   408 Mbits/sec   13             sender
[ 11]   0.00-10.01  sec   486 MBytes   408 Mbits/sec                  receiver
[ 13]   0.00-10.01  sec   486 MBytes   407 Mbits/sec    0             sender
[ 13]   0.00-10.01  sec   486 MBytes   407 Mbits/sec                  receiver
[ 15]   0.00-10.01  sec   486 MBytes   407 Mbits/sec    0             sender
[ 15]   0.00-10.01  sec   486 MBytes   407 Mbits/sec                  receiver
[ 17]   0.00-10.01  sec   486 MBytes   408 Mbits/sec    0             sender
[ 17]   0.00-10.01  sec   486 MBytes   408 Mbits/sec                  receiver
[ 19]   0.00-10.01  sec   486 MBytes   408 Mbits/sec    0             sender
[ 19]   0.00-10.01  sec   486 MBytes   408 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  3.80 GBytes  3.26 Gbits/sec   27             sender
[SUM]   0.00-10.01  sec  3.80 GBytes  3.26 Gbits/sec                  receiver


If I do something with this interface such as reassign it to another interface or even reassign the vlan to another interface the speeds will return to the expected state.

Now if I reboot ALL interfaces will be in this slower state until moved around but they will retain this fixed state on the touched interfaces until the next reboot.

Hardware CRC, Hardware TSO, and Hardware LRO are all disabled and VLAN Hardware Filtering is set to Leave default. All my system tuneables are set to default right now.

I am so confused and not sure what is going on but I need help and not sure what else to do. Willing to provide any additional debug information needed to help troubleshoot and test.

Thanks!