OPNsense Forum

English Forums => Hardware and Performance => Topic started by: cwriter on February 28, 2021, 12:30:38 pm

Title: [NIC] Intel X550 SR-IOV with VLAN
Post by: cwriter on February 28, 2021, 12:30:38 pm

Hi

I have a somewhat special use case:
I need to split a single port of an Intel X550-T2 into 5 different subinterfaces, 4 of which are carrying a VLAN tag with all different MAC addresses.

There have been a couple of hickups on the ride. It's possible to create VLAN-subinterfaces on ix0, yes. But then, I cannot have unique MACs (FreeBSD sets one MAC for parent and all subinterfaces).
I can get 2 nets going by creating a bridge, which allows getting 2 different MACs. But since you cannot have multiple bridges, this does also not work.

Then I remembered that the X550-T2 allows splitting one Physical Device into multiple Virtual Functions by using SR-IOV. I got this to work with different MACs by using iovctl to split the Device. After splitting, the parent (ix0) is unresponsive, but the created interfaces (ixv0 through ixv4) work fine.
I did the splitting by compiling the ix and ixv drivers, putting

Code: [Select]

if_ix_updated_load="YES"
if_ixv_updated_load="YES"
into /boot/loader.conf.local, copying the compiled drivers (with SRIOV_ENABLE=1) into /boot/modules/ and adding a syshook script to run iovctl.

However... When creating a VLAN-Subinterface, ix0 and all (VF) children start dropping packets and do not respond anymore.

Now: I know that SR-IOV was primarily intended for passing them to a guest vm, which I am not doing. I'm doing this only as a workaround to get unique MAC addresses.
I suspect that there are issues due to loading the ixv driver which should not allow setting VLANs nor enabling promiscous mode due to security concerns. However, when creating a VLAN subinterface on ixv1, ixv0 drops connection. So I suspect some kind of underlying bug either in the driver or in the error handling when creating a subinterface.

Has someone else experienced issues with X550, SR-IOV and VLANs? I've found conflicting information on the web about this, e.g.: https://netbsd.itsx.net/freebsd+bhyve+sriov.html


Thanks

Title: Re: [NIC] Intel X550 SR-IOV with VLAN
Post by: Ramalama on June 20, 2023, 06:37:21 pm

This is pretty old, but you need to disable spoofchk simply in the driver.

I don't know how to do this in freebsd, but on Linux it's simply "ip link set enp35s0f0 vf 0 vlan 0 spoofchk off"
vlan=0 disables vlan filtering on the vf device, which you need if you use one vf with multiple vlans. And disabling spoofchk is probably not needed if you disable vlan filtering, but i would do it anyway.

But if you split anyway your pf to multiple vf devices, why not making simply 5 vf devices for example, one for each vlan?
Then you won't need to diable spoofchk or vlan filtering, but you need to tell the driver which vf uses what vlan.
In linux that would be:
ip link set enp35s0f0 vf 0 vlan 25
ip link set enp35s0f0 vf 1 vlan 26
ip link set enp35s0f0 vf 2 vlan 27
and so on...

However, i don't know the freebsd commands for that, but at least that's hopefully a step forward to a solution.
And yes this thread is very old, but in case someone needs that info, then there is at least a solution and not just an useless thread.

Cheers