Messages

It's been working perfectly for me for a few days now... if it doesn't, I hear it immediately from my wife or the kids  ;)

Homeassistant runs on the same Proxmox/NIC as the OpnSense cluster slave and it can still reache the separate IoT LAN  when the master OpnSense is down for maintenance...
And I'm getting 4Gbit/s iperf between the 2 OpnSenses without any performance tunings and without any resends.

The only downside is the litle higher power consumption becauso of max C3 CPU state instead of C8 with the older X710.

One quirk? If I do a snapshot in Proxmox of the main OpnSense, it freezes it for a few seconds and for this time a HA failover happens. But I don't know if this is related to SR-IOV or the qemu guest of OpnSense not supporting all features.

Why not...

I myself use my E810 (from ebay) and live with the fact that my Proxmox/OpnSense SR-IOV firewall/HA-Cluster node with VLAN tagging in HW now requires 30 watts instead of 22 watts on average. For home useage not too bad, only 20$ electricity bill more per year  >:(

I have a consumer Intel H770 board with great IOMMU groups (every device + its functions separate) and i3-13100 only. But I've never had starting issues on VF NIC's with Win11, Ubuntu and OpnSense... and no need for acs override or other hacks

Note that I have no starting issues on both X710 and E810

My issues are:
X710 = CARP does not work properly (only) with VM's on same NIC
E810 = CARP works but C8 state not reachable for low power consumption... X710 can do this, E810 has disabled ASPM

Hi,

It's not an issue "does not start"... OpnSense on Proxmox works great also with SR-IOV (I've updated to Proxmox 8.2.2 last weekend and it runs great). If it does not start, you probably have to disable secure boot in the "Guest BIOS" => that was my issue when I installed OpnSense on Proxmox the first time ;D

Your error message "smells like" none unique IOMMU groups...

It's an issue with Intel virtual function network interfaces and high availability virtual IP addresses that uses CARP. The issue is that CARP needs a second MAC address and the packet flow inside the Intel driver has some "issues with this by design" on X710 NIC's. That's why it is possible to ping the CARP IP from outside (from another client/PC) but not if the client runs "on the same physical NIC" with another virtual function network device on the same physical card.

As I figured out (and also this link tells us https://forum.proxmox.com/threads/issues-with-sriov-based-nic-passthrough-to-firewall.66392/) it's needed to define "vf-true-promisc-support on" on the Proxmox host on the first NIC interface + promisc is needed to be set within the guest (in our case OpnSense / I think for CARP OpnSense enables promisc anyway?). With this settings and a newer Intel E810 card all works... but it still doesn't work on older X710 Intel NIC's.

Regards

NVM update of my X710 to current version 9.4 (or something) did also not work in my tests.

here the output of my E810

Code Select Expand

root@proxmox:~# ethtool -k enp7s0f0np0
Features for enp7s0f0np0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: on
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off
rx-fcs: off
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off
rx-vlan-stag-filter: on
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

Code Select Expand

root@proxmox:~# ethtool --show-priv-flags enp7s0f0np0
Private flags for enp7s0f0np0:
link-down-on-close     : off
fw-lldp-agent          : off
vf-true-promisc-support: on
mdd-auto-reset-vf      : off
vf-vlan-pruning        : off
legacy-rx              : off

There are far less priv-flags???

[V]

yes I mean virtual function... VT is "Virtualization Technology"

yes the vf-true-promisc flag must be set on the Proxmox host... before the first VM is started (you will get an error if a VM is running that uses such a VF NIC when you try to set "vf-true-promisc-support on"). I think before or after "echo" doesn't matter. With ethtool -k IFNAME you can see the current flags on your interface... and also yes (as I understood) this flags are for all virtual function network adapers on this IF.

I think it will also not work on your X710 card...  :(

[DID NOT WORK!]

I have tested my findings also on an Intel X710-DA2 NIC... and it did NOT work. Even after an NVM update to the newest version.

The following statements applied on the X710 NIC:

Code Select Expand

echo 8 > /sys/class/net/enp7s0f0/device/sriov_numvfs
ethtool --set-priv-flags enp7s0f0 vf-true-promisc-support on
ip link set enp7s0f0 vf 0 mac 76:9e:17:83:00:00
ip link set dev enp7s0f0 vf 0 trust on
ip link set dev enp7s0f0 vf 0 spoofchk off
ip link set enp7s0f0v0 promisc on

DID NOT WORK!

Shutdown the test rig, swapped back to E810 and applied the same statements (search replace enp7s0f0 with enp7s0f0np0):

Code Select Expand

echo 8 > /sys/class/net/enp7s0f0np0/device/sriov_numvfs
ethtool --set-priv-flags enp7s0f0np0 vf-true-promisc-support on
ip link set enp7s0f0np0 vf 0 mac 76:9e:17:83:00:00
ip link set dev enp7s0f0np0 vf 0 trust on
ip link set dev enp7s0f0np0 vf 0 spoofchk off
ip link set enp7s0f0v0 promisc on

Started OpnSense (LAN on VT 0) + the Win11 test VM (on VT 3)... and it all works!

I think I got it! I managed to ping the CARP IP (Intel E810 NIC) of a test OpnSense VM firewall and even open the management web GUI over these CARP IP from within an Ubuntu + Win11 VM running on the same Proxmox host also useing virtual adapters on the same NIC and same PF... Do you want to know what I've done ;D Did some Google research and tried out:

Code Select Expand

ethtool --set-priv-flags enp8s0f1np1 vf-true-promisc-support on

On the Proxmox host before starting any VM (enp8s0f1np1 is my PF of all the test VM's with the virtual adapters).

Can you test this too on your setup?

One downside is that this setting is global for all VT's on this NIC... but the trust on could be off on all other VT's and just be on for the OpnSense VT.

No special driver, just the default that comes with newest proxmox 8.1.5

Activated and configured the virtual adapter like this (defined a fix MAC, eanbled spoofing and trust):

Code Select Expand

echo 8 > /sys/class/infiniband/mlx5_1/device/sriov_numvfs
ip link set dev enp8s0f1np1 vf 7 mac xx:yy:zz:..
ip link set dev enp8s0f1np1 vf 7 spoofchk off
ip link set dev enp8s0f1np1 vf 7 trust on

I haven't had any issues setup or use these virtual NIC within the OpnSense VM or access the normal LAN IP to configure a test CARP IP...

It left me no peace... I did a "quick" test with a ConnectX4.

But I have a different problem with this setup, I can't achieve a MASTER CARP state on a Mellanox VT interface. It looks as if with Mellanox cards the CARP requests are sent to themselves and therefore always a "better" master is available. I am currently unable to obtain a MASTER CARP IP even with just one OpnSense instance running.

Getting the following log messages:

Code Select Expand

<6>carp: 8@mce0: MASTER -> BACKUP (more frequent advertisement received)

So multiple MAC's on the same SR-IOV virtual adapter does not work at all???

Strange, a virtual IP of type "IP Alias" works well also from such clients  :-\
Whereby these have the same MAC address as the real LAN NIC IP

Maybe I can test it with a Mellanox ConnectX4, but unfortunately certainly not before easter...

Yes, I plan a similar setup and run into the same issue during testing.

Proxmox host, newest OpnSense version running with a CARP IP on LAN... all clients on my network can reach this IP, the Proxmox host can ping it too, but no other VM client on this host that uses virtual function NIC's can ping the CARP IP.

The only thing that works from such a client is the ARP broadcast for the CARP IP, so the client knows the MAC address of the CARP IP but after that no packages received by Opnsense (traffic to "real" IP of the OpnSense no issues!).

In my case, I have Intel E810... i think it's an Intel iavf driver issue as the 710 and 810 cards uses the same VF driver.

Hi

Have you fixed this problem in the meantime?

Regards

As a heads up, usually most providers allow you to switch devices if you call them. You simply need to give them your MAC address and it works.

Not the provider I have   >:(

My solution right now is: I have changed the MAC address within the eprom of the WAN NIC. There exists a tool called eeupdate.exe (can be found via Google, it's not official by Intel) which can do this on many Intel NIC's. Now with the "physical" changed MAC, I don't need to spoof it any longer and so no flapping...