SR-IOV with intel X550-T2 and OPNsense on Proxmox [solved]

[binaryanomaly]

Hi,

I'm having inconsistent to erratic behavior with OPNsense as a VM guest in combination with SR-IOV.
I have SR-IOV enabled on the Proxmox host and also for a Debian guest - the latter as a verification.
The throughput increase on OPNsense with suricata enabled is an impressive factor 3-4 which makes SR-IOV worthwhile.

But the OPNsense guest sometimes doesn't want to run at all with the VF interface, sometimes runs fine for hours  and then the VF interfaces suddenly stops working for OPNsense - sometimes after a reboot or just out of the blue, for no obvious reason.

I have no clue what the cause of this inconsistent behavior is and I do not see anything meaningful in dmesg output or in /var/log/system.log or on the host besides sudden link state changes of the VF interface in the guest VM only.

On the Debian guest though everything keeps running smoothly all the time so the problem seems only to be related to the OPNsense guest.

Is this a known upstream FreeBSD issue or should this work in general with OPNsense?
Where could I look for helpful log data - dmesg and system.log have not proofed to be very helpful so far.

Thanks

Edit: Wrong forum this is on 21.1

[Patrick M. Hausen]

You might be better off with PCI-e passthrough if you can spare the interfaces. I don't know about the current state of support for SR-IOV in FreeBSD but OPNsense currently runs on an outdated version of FreeBSD with backported security and bug fixes. You could try running a stock FreeBSD 12.2 in your Proxmox environment and if that is stable, wait for the next major release of OPNsense early 2022. It will catch up with FreeBSD by that time.

If FreeBSD 12.2 exhibits the same problems, you should file bug reports with FreeBSD.
 
HTH,
Patrick

[binaryanomaly]

Thanks for your response. I may give 13/12.2 a try and observe.

SR-IOV would be just too nice, to give it up too early. And I'm afraid of running into the same issues passing along the whole card.

I'm mostly irritated by not finding any errors in dmesg and system.log

[binaryanomaly]

I think I identified a pattern and a rather insane workaround for the time being:

It seems that the VF interface that is present at OPNsense boot time quickly works (a few seconds) but then immediately after this gets somehow burned and becomes defunct.

My approach to mitigate this is atm to expose two VF PCI cards to OPNsense and after boot switch to the other one - insane but so far it works until next reboot when this is required again

in dmesg it looks like this:

Code: [Select]

ixv0: permanently promiscuous mode enabled
ixv0: link state changed to DOWN
ixv0: link state changed to UP
ixv0: link state changed to DOWN
ixv0: link state changed to UP
ixv0: link state changed to DOWN
ixv0: link state changed to UP
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled

--> Switching interface to ixv1 here

ixv1: link state changed to DOWN
ixv1: link state changed to UP
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled

Assumption 1
Could it be related to promiscuous mode?
I see it happening for ixv0 but never for ixv1 "ixv0: permanently promiscuous mode enabled"
I assume this is triggered through the respective suricata setting and may not be executed properly again after changing the WAN interface assignment and therefore leave it working?

Assumption 2
Could this maybe be related to an old version of the intel driver being used with OPNsense?
Is it possible to pull in a newer version?

[Patrick M. Hausen]

Is it possible to pull in a newer version?

That will happen automatically with the switch from HardenedBSD to stock FreeBSD in 2022.

[binaryanomaly]

That will happen automatically with the switch from HardenedBSD to stock FreeBSD in 2022.

That I understood. But isn't it possible to pull are more recent driver and build a module as with Linux?
Or is with FreeBSD everything hardwired into the kernel?

[binaryanomaly]

Seems it's possible according to here:
https://downloadmirror.intel.com/14688/eng/README_3.3.24.txt

But I also stumbled across this:

Some notable limitations of the VF environment:
  * For security reasons, the driver is never permitted to be promiscuous, therefore a tcpdump will not behave the same with the interface.

Which seems to confirm my assumption related to promiscuous mode.
--> That means it will likely only work when passing through the whole device.

Thanks for your help nevertheless.

[Patrick M. Hausen]

That will happen automatically with the switch from HardenedBSD to stock FreeBSD in 2022.

That I understood. But isn't it possible to pull are more recent driver and build a module as with Linux?
Or is with FreeBSD everything hardwired into the kernel?

It is entirely possible. Just not a thing "common users" regularly do. You would need to setup a HardenendBSD build environment, checkout the sources and merge the Intel driver sources, then build the module.

[binaryanomaly]

Ok got it, thanks.

Now that I discovered the VF promiscuous mode limitation the only option that will work is indeed as you initially suggested to hand through the whole device/port.