I have a NAS, and a computer, with ASUS XG-C100C 10 Gbit NICs.
In between I have a OpnSense router with two Intel X550-T2 10 Gbit NICs.
When transferring a big file from NAS to computer I get about 6.8 - 7.2 Gbit, according to the Windows 11.
The NAS has an SSD Raid that can deliver 10 Gbit of data without a problem.
I was hoping to increase this to closer to 10 Gbit by setting the MTU in OpnSense and jumbo frame in Windows 11.
If I set the MTU to 9000 on the Intel NIC in OpnSense GUI the connection to my computer drops out back and fourth. If I ping the router 4-5 pings get through and 4-5 pings then doesn't.
If I do a ipconfig, while MTU is set, it says that the MTU is 1500:
ix2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN10GB1 (opt2)
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,NOMAP>
ether xxxx
media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
If I remove the MTU value in the GUI and set it using the command line ifconfig ix2 mtu 9000 up it shows up as 9000 if checked with ifconfig but it gets reset back to 1500 after a while and I don't think it actually worked since I can't verify using ping -f -l 9000 ... even after setting the computer NIC jumbo frame.
Any suggestions to get this to work?
pciconf -lvc
ix2@pci0:4:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 subvendor=0x8086 subdevice=0x0001
vendor = 'Intel Corporation'
device = 'Ethernet Controller 10G X550T'
class = network
subclass = ethernet
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 64 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO
max read 512
link x4(x4) speed 8.0(8.0) ASPM disabled(L0s/L1)
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 915049ffff200000
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x0180, VF RID Stride 0x0002
VF Device ID 0x1565
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1b0] = ACS 1
ecap 0018[1c0] = LTR 1
ecap 0019[1d0] = PCIe Sec 1 lane errors 0
At least with the Linux drivers, it seem like that adapter cannot set MTU for each interface individually:
https://doc.dpdk.org/guides-21.11/nics/ixgbe.html
Since you use a T2: Did you set the MTU on both cards the same?
I tried now to set the MTU to 9000 for both ports on both NICs, using the GUI.
Same problem as before.
I then tried to set it using command line:
ifconfig ix0 mtu 9000 up & ifconfig ix1 mtu 9000 up & ifconfig ix2 mtu 9000 up & ifconfig ix3 mtu 9000 up &
Still same problem as before.