Issue with MTU on 10G interface

[mrFFM]

I have an issue with the MTU on a 10G interface and would hope someone could give me a hint on how to fix this.

Context:
I am currently migrating from a DEC690 to a DEC850. Two vlans are configured on the 1GB interfaces and everything works as expected. Objective is to trunk both VLANs on the 10G interface. After setting up the new config throughput dropped significantly.

HW/SW Info:
OPNSense: DEC850 with 21.4.2
Switch: Zyxel XGS1250-12 with FW V1.00(ABWE.0)C0

OPNSense setup:
igb0: wan
igb1: vlan10 (no other networks)
igb2: vlan20 (no other networks)

igb1 and igb2 are connected to different ports on the same switch. All switch ports are member of exactly one VLAN (tagged for the uplink to opnsense, untagged for clients). Throughput is as expected for a 1G interface. The 10GE connection between DEC850 and switch is via an SFP+ DAC cable. 

During troubleshooting I found the following:
1st step:

• setup ax0 as new interface (new subnet but no vlan)
• create a new vlan (vlan40) on the switch
• add the SFP+ port and one 1GB port to the vlan40 (both untagged, old vlans are removed from the two ports).

==> All fine, good throughput


2nd step

• create a new vlan on opnsense (vlan40), remove the previous config on ax0 and assign the vlan to ax0
• on the switch, change the SFP+ uplink to tagged in vlan40

==> Throughput form the client to opnsense drops significantly (an iPerf to a PI drops from 940 MBits/s to around 3MBits/s)
[/list]
Error message on the OPN console:  ax0: axgbe_isc_rxd_pkt_get: Big packet 1518/1514

Checked MTU on ax0 and ax0_vlan40: both are at 1500. Once I manually set the MTU on ax0 to 1504 (ifconfig ax0 mtu 1504) all works fine.

Questions:

• Is it to be expected that I manually need to increase the MTU on ax0? What strikes me is that in the 1GB setup the MTU on igb1 and igb1_vlan10 is at 1500 without any issues, but I have to increase the MTU on ax0 to create space for the vlan tag. If not, what else should I be looking into?
• If increasing the MTU on ax0 is indeed the appropriate solution, how can I best configure the value persistently (i.e. not overwritten during reboot or upgrades).

Many thanks for your support!

[va176thunderbolt]

Not sure if this helps or not, but I have an intel X540-t2 and a X550-t2 (both use the ix driver), and both defaulted to 1500 mtu's. Both are configured with one untagged interface, and one with tagged and untagged interfaces configured.

[tuto2]

Hi,

As soon as you increase the MTU on the ax0 interface by four bytes, does your throughput stabilize as expected?

Logging error messages may significantly affect performance, especially if they are on a per-packet basis.

Furthermore, could you try enabling hardware VLAN tagging in interfaces -> settings?. Though this logically shouldn't be necessary given the expected systems' capability of dealing with both scenarios, I think an incomplete assertion is being done in the driver code.

Cheers,

Stephan

[mrFFM]

Many thanks to both of you for your suggestions!

Not sure if this helps ...

It helped, as it reassured me that one shouldn't have to change the MTU. It also triggered a thinking process: changing the MTU would likely cause issues as soon as there was untagged traffic on the same interface

As soon as you increase the MTU on the ax0 interface by four bytes, does your throughput stabilize as expected?

Logging error messages may significantly affect performance, especially if they are on a per-packet basis.

Furthermore, could you try enabling hardware VLAN tagging in interfaces -> settings?. Though this logically shouldn't be necessary given the expected systems' capability of dealing with both scenarios, I think an incomplete assertion is being done in the driver code.

Yes, the system is perfectly fine with increased MTU. I couldn't find HW VLAN tagging in interfaces -> settings (or would this be included in VLAN HW Filtering?) but I enabled it with "ifconfig ax0 vlanhwtag" and reverted the MTU to 1500. Indeed, this also solves the issue. I also tried to get rid of the logging but wouldn't be able to see where (axgbe_debug_level is 0 and Interfaces->Settings->Log level is standard).

Sorry for the novice question, but how could I best make the HW VLAN tagging persistent? Happy to check what's happening with logging reduced if you could please give me a hint on how to accomplish that.

[tuto2]

Hi,

I couldn't find HW VLAN tagging in interfaces -> settings (or would this be included in VLAN HW Filtering?)

Sorry for the novice question, but how could I best make the HW VLAN tagging persistent? Happy to check what's happening with logging reduced if you could please give me a hint on how to accomplish that.

Yes, I meant VLAN HW Filtering, apologies. But since you enabled it via ifconfig and it has been stable since, my suspicion has been confirmed.

I also tried to get rid of the logging but wouldn't be able to see where (axgbe_debug_level is 0 and Interfaces->Settings->Log level is standard).

The logging for this cannot be removed, since it is a hardcoded error in the driver, I woudn't mess with the debug setting since it will complete clog up your system with logs on full load.

I will submit a bugfix soon that should fix this issue, until then, the two following workarounds can be used:

- Increasing MTU by 4 bytes to 1504
- Enabling VLAN HW filtering.

Keep in mind that there is a reason VLAN HW filtering is disabled by default on OPNsense, as problems have arisen with the Intel igb driver in the past. Furthermore, I am unsure of what version of OPNsense you are running, the latest versions support per-interface settings such as VLAN HW filtering and receive checksum offloading, the business edition however does not yet support this. I would recommend keeping your MTU at 1504 for now.

If your edition does support per-interface hardware settings, I would recommend enabling VLAN HW filtering.

Cheers,

Stephan