1
High availability / Re: CARP IP not pingable from other SR-IOV virtual function on same host
« on: April 29, 2024, 04:51:36 pm »Hi,
It's not an issue "does not start"... OpnSense on Proxmox works great also with SR-IOV (I've updated to Proxmox 8.2.2 last weekend and it runs great). If it does not start, you probably have to disable secure boot in the "Guest BIOS" => that was my issue when I installed OpnSense on Proxmox the first time
Your error message "smells like" none unique IOMMU groups...
It's an issue with Intel virtual function network interfaces and high availability virtual IP addresses that uses CARP. The issue is that CARP needs a second MAC address and the packet flow inside the Intel driver has some "issues with this by design" on X710 NIC's. That's why it is possible to ping the CARP IP from outside (from another client/PC) but not if the client runs "on the same physical NIC" with another virtual function network device on the same physical card.
As I figured out (and also this link tells us https://forum.proxmox.com/threads/issues-with-sriov-based-nic-passthrough-to-firewall.66392/) it's needed to define "vf-true-promisc-support on" on the Proxmox host on the first NIC interface + promisc is needed to be set within the guest (in our case OpnSense / I think for CARP OpnSense enables promisc anyway?). With this settings and a newer Intel E810 card all works... but it still doesn't work on older X710 Intel NIC's.
Regards
I have been running OPNsense and other VMs with SR-IOV for years now, no problems. It's only kernel 6.8 with the X710 interface preventing any of my VMs (Linux or OPNsense) from starting. It's a Supermicro EPYC board with full IOMMU support, no hacks required.
Older Intel 10G card works fine, too.
I have ordered an E180 adapter now, you not having any issues with that one is a good starting point.