Multi-WAN gateway monitoring dashboard problem with virtual adapaters

[patrick3000]

I have a multi-WAN setup with gateway monitoring in OPNsense 24.7.6 that I run virtually on Truenas SCALE.

It works properly if I pass through the two WANs (called "WAN" and "WAN2") using PCIE pass-through. However, if I virtualize either of the WANs and pass it through as a virtual adapter, then the dashboard always shows a green dot, which is supposed to indicate that the interface is up, even when the interface is down. (Please note that this appears to be a dashboard problem only, because gateway monitoring appears to still actually work and fail over to whichever WAN is still up.)

As I recall, this problem with the dashboard and virtual adapters subject to gateway monitoring did not exist in earlier versions of OPNsense, such as 24.2, which was the last time I passed through virtual adapters. So, it appears to be a new problem, and I suspect is a bug, in 24.7.

Does anyone know how to fix this dashboard problem with gateway monitoring in version 24.7? Alternatively, does anyone have any experience or feedback with this problem?

[klosz007]

Hi,

What do you mean with "virtual adpater" here ?
 
I have not been running any VMs on TNS. But I run my OPNsense instances on PVE, albeit not with passthrough PCIe NICs. I do have PCIe passthrough devices on other VMs on PVE though (my TNS is a VM on PVE and has passthorugh NICs).

My expierence is that if you passthrough entire network adapter (=port) to a VM then its physical link state is properly indicated inside the VM - it replicates state of the physical link. But in fact then it is not "a virtual adapter" (as you called it) but just a physical adapter (physical function = PF) that is passed through/redirected to a VM. In other words VM sees a physical NIC.

If you are using SR-IOV on the other hand then you are redirecting a "virtual" instance of the adapter (VF - virtual function), not a PF.

My experience says that link state for VFs is always up, no matter whether link state of PF is up/down. Maybe that behavior depends on NIC type or its driver but makes sense to me. Even if link is down in PF then you may have other VFs on the same PF and they will still be able to communicate to each other via an integrate switch inside SR-IOV capable NIC. It would not be possible if VF link went down when PF link goes down.

So my question is what are you actually redirecting to VM if you are calling it "virtual adapter" ? Entire adapter (PF) or VF of SR-IOV ?

[patrick3000]

Proxmox (PVE) and Truenas Scale are both KVM on top of Debian, so for the purpose of virtualization, they are very similar.

By "virtual adapters," I am referring to the feature in Linux, FreeBSD, and other Unix-like operating systems that allows adapters to be passed to a guest VM, where they appear as "vnet1," "vnet2," etc. This is different from PCIE pass-through, which is more like the VM having full control of the adapter as though it were the host. Both have advantages and disadvantages. The reason I am trying to use virtual adapters, rather than what I currently use which is PCIE pass-through, for WAN and WAN2, is that I would like to eventually run WAN and WAN2 through a switch as VLANs, and then pass each VLAN to OPNsense as a virtual adapter.

Gateway monitoring with virtual adapters does not reflect properly in the dashboard in OPNsense 24.7.6, however, because not only does the dashboard show WAN (or WAN2) as up (in green) even when it's down, but it shows packet loss as 0.0%.

[klosz007]

So with "virtual" you mean VirtIO paravirtualized NICs plugged into Linux bridge on KVM hypervisor side ? In such case I guess you meant "vtnet" not "vnet" ? (that's how these are visible in FreeBSD so in OPNsense). I use these on my PVE OPNsense VMs.

I have never paid attention to how they work in OPNsense but my understanding will be same as with PCIE passthrough with SR-IOV - driver always reports their link state to be up and will not replicate physical NICs state that is connected to the underlying Linux bridge. So they will always show up as green in OPNsense, unless you forcibly/manually put link down on such virtualized NIC in KVM VM's configuration.

I'm surprised there were any changes around 24.7.6 in this regards.

Gateway state reporting will still work fine becuase if physical link dies, you are not able to ping gateway anymore (assuming you are monitoring gateway IP behind the physical cable but not another VM's IP for example).

If VirtIO link went down when physivcal link of the NIC connected to the bridge went down then you would lose ability to talk to another VMs in the same bridge when physical link goes down. I'm guessing in most cases it is undesirable.

I saw an option to replicate physical NIC linkstate to virtualized NIC linkstate in standalone desktop hypervisors (e.g. VirtualBox) but not in KVM.
With KVM, if you want physical link state be replicated to NIC in the VM then my best guess would be to stay with PCIe passthrough and effectively reassign physical NIC from hypervisor to VM.

[patrick3000]

Yes, by virtual adapters, I mean VirtIO paravirtualized adapters, and you're correct, they appear in the OPNsense VM as "vtnet," not "vnet," which was a mistake.

I now see what you're saying that they will show as green in OPNsense even if the underlying physical adapter in the Linux host is down. However, I still think there is something wrong with the OPNsense dashboard, because it shows "Loss: 0.0%" for WAN, even when gateway monitoring has taken WAN off line due to being unable to ping the monitored IP (which is 8.8.8.8, i.e. google in my case).

Still, maybe this isn't such a big deal because gateway monitoring with fail-over still appears to actually work. It's just that the dashboard incorrectly shows 0.0% packet loss even when monitoring is unable to successfully ping.

[klosz007]

Yes, physical NIC state connected to Linux bridge does not propagate to (para)virtualized (not virtual :-) NICs. That would be undesireable in most server applications - that is where we use KVM. You want VMs always to be able to talk to each other, even if physical NIC link went down. You can always simulate link down for virtualized NIC if needed, by setting link down option for given NIC in VM's options.

Such propagation makes sense on desktop virtualization though. I'm not sure if WVware workstation has such option, VirtualBox has it for sure.

I'm trying to find a place in the dashboard you are talking about...
Packet loss is gateway's statistic, not network interface's. And it will be reported correctly because when physical NIC link goes down, you are not able to ping your test IP even if link on virtualized NIC is still up. Interfaces do not have packet loss statistic, I cannot see it. They have packet in/out, bytes in/out or errors.

[patrick3000]

Here is a screen shot of the OPNsense dashboard gateway status, which shows packet loss (currently 0.0%, which is correct, because both WAN and WAN2 are up). IP addresses have been redacted.

The way it's always worked with gateway monitoring and PCIE pass-through is that if either WAN or WAN2 goes down, the "Loss" value gradually climbs from 0% to 100%, and when it gets above, as I recall, 20% (a number which can be set somewhere), it switches the dot from green to red for that gateway, takes the interface offline, and fails over to the other gateway.

With para-virtualized adapters rather than PCIE pass-through, when WAN or WAN2 goes down, this part of the dashboard continues to show Loss as "0.0%" and never switches the dot from green to red, even though fail-over to the working interface appears to happen properly.

[klosz007]

OK, I know what you mean. I run latest version of OPNSense (24.7.6) with paravirtualized vtnet adapters (on PVE)  and I do not have this issue. When I unplug the physical network cable to DSL modem, OPNsense interface stays connected (plug symbol stays green), packet loss rises in the gateway stats and that green dot changes first to orange then to red. I do not think the issue has anything to do whether you use paravirtualized or physical/redirected NICs in your VM. I cannot recreate it here so cannot help anymore. But I believe it is some kind of misconfiguration somewhere.

[patrick3000]

Thanks for trying.

However, I have now done some additional testing, and I believe that this problem might not be an OPNsense problem. It might be a host (KVM or Truenas) problem. Here is why I think that.

I just now tried passing through a different adapter (not the one I ultimately want to use) using para-virtualization, and OPNsense handled gateway monitoring and fail-over properly. In particular, when the cable to the physical adapter was yanked, the dashboard showed packet loss rising, and then the dot changed from green to red.

This is not the adapter I ultimately want to use because it's a 1 gpbs adapter, and my internet plan is 1.25 gbps, so I need to use the other adapter (which is 10 gbps and supports n-base T, and negotiates with the modem at 2.5 gbps).

It appears, then, that I only have this problem with one particular physical adapter (the 10 gbps adapter) that I pass to OPNsense. I suspect the problem might relate to not rebooting the host server between when this adapter is used for PCIE pass-through and when it's used for para-virtualization, but I cannot easily reboot the host server for the next couple days so cannot test that theory at the moment.

In any event, it now seems unlikely that this is an OPNsense problem. However, I will leave this thread up in case anyone finds it useful. And again, thanks for your help trouble-shooting.