OPNsense Forum
English Forums => High availability => Topic started by: Simon42 on February 18, 2024, 01:27:11 pm
-
So let me start with a little diagram to hopefully make this better understandable:
(https://i.imgur.com/m5RzHCt.png)
I have 2 proxmox hosts each running an opnsense vm for HA.
Both of these hosts have an intel xl710 installed, but pve-router has the full card PCI-passthroughed and on pve-main I created multiple SR-IOV virtual functions(VFs) on the host and just used PCI-passtrough on one of the virtual function's pci-device. pve-main also has some other vms (on other VF) handling other services.
Normally the pve-router (master opnsense) handles all the traffic and everything is fine. But when this one fails, the main server(slave opnsense) should take over routing for the time.
So when OPNsense2 becomes CARP-master, here comes the Problem:
The CARP IP is not pingable from other vms/VFs on the same host. or to be more specific:
vm1 (10.10.110.200) can't ping carp (10.10.110.1)
but vm1 can ping directly to 10.10.110.3
checking with a client outside(10.10.110.40), the client can ping both (.3 AND .1), so the carp is should theoretically be setup fine?
But something seems to go wrong when the traffic is heading to the carp ip on the same host (and this one is using sr-iov VFs - as when opnsense1 is master (which is not using a VF) everthing works).
Some more debugging I already did:
- Started a Interfaces: Diagnostics: Packet Capture on the interface (including the Promiscuous checkbox checked) and tried vm -> carp ping again - unfortuneately nothing: no traffic seems to reach the firewall at all (capture works because pinging .3 from vm or .1 from client the capture shows traffic)
- Checked the arp table of 10.10.110.1 on both vm1 and client: both correctly point to the virtual carp mac (00:00:5e:00:01:0a)
PS:
- The VF has spoof checking off, trust on
Anyone any idea what could go wrong here?
-
Hi
Have you fixed this problem in the meantime?
Regards
-
Unfortunately not (yet).
Do you have the same / a similar problem also?
If yes, please keep me updated here if you find something.
-
Yes, I plan a similar setup and run into the same issue during testing.
Proxmox host, newest OpnSense version running with a CARP IP on LAN... all clients on my network can reach this IP, the Proxmox host can ping it too, but no other VM client on this host that uses virtual function NIC's can ping the CARP IP.
The only thing that works from such a client is the ARP broadcast for the CARP IP, so the client knows the MAC address of the CARP IP but after that no packages received by Opnsense (traffic to "real" IP of the OpnSense no issues!).
In my case, I have Intel E810... i think it's an Intel iavf driver issue as the 710 and 810 cards uses the same VF driver.
-
Would be interesting to know if we could find someone who got it working with a different nic / driver and vfs.
So we could be more certain its actually a driver problem and not something else.
If so, maybe contacting Intel support then, but getting them to understand this honestly quite complex problem, acknowledging its a problem in their driver and fixing it is another story....
-
Maybe I can test it with a Mellanox ConnectX4, but unfortunately certainly not before easter...
-
Strange, a virtual IP of type "IP Alias" works well also from such clients :-\
Whereby these have the same MAC address as the real LAN NIC IP
-
No hurry, would be great if you could test and post your findings.
Yeah IP Alias as you said has same MAC.
I guess the vf only has one MAC assigned (and not the CARP). So it does not receive Carp packages (although promiscuous mode is enabled). And for some reason all this applies to packets within the same nic. (So maybe its indeed a driver issue with packet routing between different vfs - as the traffic should not leave the nic)
-
It left me no peace... I did a "quick" test with a ConnectX4.
But I have a different problem with this setup, I can't achieve a MASTER CARP state on a Mellanox VT interface. It looks as if with Mellanox cards the CARP requests are sent to themselves and therefore always a "better" master is available. I am currently unable to obtain a MASTER CARP IP even with just one OpnSense instance running.
Getting the following log messages:
<6>carp: 8@mce0: MASTER -> BACKUP (more frequent advertisement received)
So multiple MAC's on the same SR-IOV virtual adapter does not work at all???
-
Interesting...
What driver / settings do you use with your connectx4?
As (in my opinion at least) dealing with mellanox drivers is quite annoying (even more than this "little" problem with the intel ones here).
For context: i had connectx 3 fcbt before but just could not get anything to work with those ... So I eventually gave up and bought intel ones.
-
No special driver, just the default that comes with newest proxmox 8.1.5
Activated and configured the virtual adapter like this (defined a fix MAC, eanbled spoofing and trust):
echo 8 > /sys/class/infiniband/mlx5_1/device/sriov_numvfs
ip link set dev enp8s0f1np1 vf 7 mac xx:yy:zz:..
ip link set dev enp8s0f1np1 vf 7 spoofchk off
ip link set dev enp8s0f1np1 vf 7 trust on
I haven't had any issues setup or use these virtual NIC within the OpnSense VM or access the normal LAN IP to configure a test CARP IP...
-
I think I got it! I managed to ping the CARP IP (Intel E810 NIC) of a test OpnSense VM firewall and even open the management web GUI over these CARP IP from within an Ubuntu + Win11 VM running on the same Proxmox host also useing virtual adapters on the same NIC and same PF... Do you want to know what I've done ;D Did some Google research and tried out:
ethtool --set-priv-flags enp8s0f1np1 vf-true-promisc-support on
On the Proxmox host before starting any VM (enp8s0f1np1 is my PF of all the test VM's with the virtual adapters).
Can you test this too on your setup?
One downside is that this setting is global for all VT's on this NIC... but the trust on could be off on all other VT's and just be on for the OpnSense VT.
-
I have tested my findings also on an Intel X710-DA2 NIC... and it did NOT work. Even after an NVM update to the newest version.
The following statements applied on the X710 NIC:
echo 8 > /sys/class/net/enp7s0f0/device/sriov_numvfs
ethtool --set-priv-flags enp7s0f0 vf-true-promisc-support on
ip link set enp7s0f0 vf 0 mac 76:9e:17:83:00:00
ip link set dev enp7s0f0 vf 0 trust on
ip link set dev enp7s0f0 vf 0 spoofchk off
ip link set enp7s0f0v0 promisc on
DID NOT WORK!
Shutdown the test rig, swapped back to E810 and applied the same statements (search replace enp7s0f0 with enp7s0f0np0):
echo 8 > /sys/class/net/enp7s0f0np0/device/sriov_numvfs
ethtool --set-priv-flags enp7s0f0np0 vf-true-promisc-support on
ip link set enp7s0f0np0 vf 0 mac 76:9e:17:83:00:00
ip link set dev enp7s0f0np0 vf 0 trust on
ip link set dev enp7s0f0np0 vf 0 spoofchk off
ip link set enp7s0f0v0 promisc on
Started OpnSense (LAN on VT 0) + the Win11 test VM (on VT 3)... and it all works!
-
Thanks for reporting back your findings.
This vf-true-promisc was actually something I had left on my to-do list of things to still try out when I got time again to investigate this further, as I found it in the nvm changelogs.
But for now didn't really know where i had to apply this (as in tge opnsense vm i had no ethtool)
Am I reading this correctly?:
So I have to do this on the proxmox host after creating the vfs/ runing echo, but before starting any vms using the vfs? And this than applies to all vfs.
I will hope to try this out in the next couple of days and report back.
PS: just to be sure: with "VT" you are talking about virtual functions? ip link calls them also "vf". What does "VT" stand for? Or am I missing something?
-
yes I mean virtual function... VT is "Virtualization Technology"
yes the vf-true-promisc flag must be set on the Proxmox host... before the first VM is started (you will get an error if a VM is running that uses such a VF NIC when you try to set "vf-true-promisc-support on"). I think before or after "echo" doesn't matter. With ethtool -k IFNAME you can see the current flags on your interface... and also yes (as I understood) this flags are for all virtual function network adapers on this IF.
I think it will also not work on your X710 card... :(
-
thanks for the info.
did some quick testing today:
enabled it with ethtool --set-priv-flags enp9s0 vf-true-promisc-support on
but does not seem to work unfortunately :(. Still ping not even reaching opnsense packet capture.
but still have some more testing left todo. (your comments don't let me hope, but lets see... ;)): still need to do an nvm update, ...
PS: ethtool -k does not show these priv-flags:
root@pve:~# ethtool -k enp9s0
Features for enp9s0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
but i got them using
root@pve:~# ethtool --show-priv-flags enp9s0
Private flags for enp9s0:
MFP : off
total-port-shutdown : off
LinkPolling : off
flow-director-atr : on
veb-stats : off
hw-atr-eviction : off
link-down-on-close : off
legacy-rx : off
disable-source-pruning : off
disable-fw-lldp : off
rs-fec : off
base-r-fec : off
vf-vlan-pruning : off
vf-true-promisc-support: on
maybe you could post the output of both of these commands for you E810, so we could compare if there are maybe some more interesting (enabled by default) flags on the E810 :)
-
NVM update of my X710 to current version 9.4 (or something) did also not work in my tests.
here the output of my E810
root@proxmox:~# ethtool -k enp7s0f0np0
Features for enp7s0f0np0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off
rx-fcs: off
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off
rx-vlan-stag-filter: on
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
root@proxmox:~# ethtool --show-priv-flags enp7s0f0np0
Private flags for enp7s0f0np0:
link-down-on-close : off
fw-lldp-agent : off
vf-true-promisc-support: on
mdd-auto-reset-vf : off
vf-vlan-pruning : off
legacy-rx : off
There are far less priv-flags???
-
Sorry for taking so long, but just found time to test again today.
Unfortunately, no success even after updating to 9.40.
Found this older thread. May be interesting: https://forum.proxmox.com/threads/issues-with-sriov-based-nic-passthrough-to-firewall.66392/
It talks about using iavf driver. (current OPNsense is already doing that by default as far as I can see)
And VLAN filters, which I don't use.
Unfortunately, no actual solution was posted there.
-
Does Proxmox kernel 6.8 also keep your hosts from starting?
This is my X710, stock or latest Intel drivers, no luck...
Firmware has requested this device have a 1:1 IOMMU mapping, rejecting configuring the device without a 1:1 mapping. Contact your platform vendor.
-
Hi,
It's not an issue "does not start"... OpnSense on Proxmox works great also with SR-IOV (I've updated to Proxmox 8.2.2 last weekend and it runs great). If it does not start, you probably have to disable secure boot in the "Guest BIOS" => that was my issue when I installed OpnSense on Proxmox the first time ;D
Your error message "smells like" none unique IOMMU groups...
It's an issue with Intel virtual function network interfaces and high availability virtual IP addresses that uses CARP. The issue is that CARP needs a second MAC address and the packet flow inside the Intel driver has some "issues with this by design" on X710 NIC's. That's why it is possible to ping the CARP IP from outside (from another client/PC) but not if the client runs "on the same physical NIC" with another virtual function network device on the same physical card.
As I figured out (and also this link tells us https://forum.proxmox.com/threads/issues-with-sriov-based-nic-passthrough-to-firewall.66392 (https://forum.proxmox.com/threads/issues-with-sriov-based-nic-passthrough-to-firewall.66392)/) it's needed to define "vf-true-promisc-support on" on the Proxmox host on the first NIC interface + promisc is needed to be set within the guest (in our case OpnSense / I think for CARP OpnSense enables promisc anyway?). With this settings and a newer Intel E810 card all works... but it still doesn't work on older X710 Intel NIC's.
Regards
-
Hi,
It's not an issue "does not start"... OpnSense on Proxmox works great also with SR-IOV (I've updated to Proxmox 8.2.2 last weekend and it runs great). If it does not start, you probably have to disable secure boot in the "Guest BIOS" => that was my issue when I installed OpnSense on Proxmox the first time ;D
Your error message "smells like" none unique IOMMU groups...
It's an issue with Intel virtual function network interfaces and high availability virtual IP addresses that uses CARP. The issue is that CARP needs a second MAC address and the packet flow inside the Intel driver has some "issues with this by design" on X710 NIC's. That's why it is possible to ping the CARP IP from outside (from another client/PC) but not if the client runs "on the same physical NIC" with another virtual function network device on the same physical card.
As I figured out (and also this link tells us https://forum.proxmox.com/threads/issues-with-sriov-based-nic-passthrough-to-firewall.66392 (https://forum.proxmox.com/threads/issues-with-sriov-based-nic-passthrough-to-firewall.66392)/) it's needed to define "vf-true-promisc-support on" on the Proxmox host on the first NIC interface + promisc is needed to be set within the guest (in our case OpnSense / I think for CARP OpnSense enables promisc anyway?). With this settings and a newer Intel E810 card all works... but it still doesn't work on older X710 Intel NIC's.
Regards
I have been running OPNsense and other VMs with SR-IOV for years now, no problems. It's only kernel 6.8 with the X710 interface preventing any of my VMs (Linux or OPNsense) from starting. It's a Supermicro EPYC board with full IOMMU support, no hacks required.
Older Intel 10G card works fine, too.
I have ordered an E180 adapter now, you not having any issues with that one is a good starting point.
-
I have a consumer Intel H770 board with great IOMMU groups (every device + its functions separate) and i3-13100 only. But I've never had starting issues on VF NIC's with Win11, Ubuntu and OpnSense... and no need for acs override or other hacks
Note that I have no starting issues on both X710 and E810
My issues are:
X710 = CARP does not work properly (only) with VM's on same NIC
E810 = CARP works but C8 state not reachable for low power consumption... X710 can do this, E810 has disabled ASPM
-
Thinking about the current state:
So because of the X710 vs E810 case, we are sure it's a Problem with the Intel drivers AND only on the X710.
Do you think contacting intel support would get us anywhere?
-
Why not...
I myself use my E810 (from ebay) and live with the fact that my Proxmox/OpnSense SR-IOV firewall/HA-Cluster node with VLAN tagging in HW now requires 30 watts instead of 22 watts on average. For home useage not too bad, only 20$ electricity bill more per year >:(
-
Yeah, maybe I look if I can also find a cheap enough E810 on ebay, now that I know this one will work for sure.
-
It's been working perfectly for me for a few days now... if it doesn't, I hear it immediately from my wife or the kids ;)
Homeassistant runs on the same Proxmox/NIC as the OpnSense cluster slave and it can still reache the separate IoT LAN when the master OpnSense is down for maintenance...
And I'm getting 4Gbit/s iperf between the 2 OpnSenses without any performance tunings and without any resends.
The only downside is the litle higher power consumption becauso of max C3 CPU state instead of C8 with the older X710.
One quirk? If I do a snapshot in Proxmox of the main OpnSense, it freezes it for a few seconds and for this time a HA failover happens. But I don't know if this is related to SR-IOV or the qemu guest of OpnSense not supporting all features.