[CLOSED] OPNsense and KVM's Virtio Network Driver

[Taomyn]

Perhaps you can post the VM config from Proxmox so we can see exactly how it's been configured?


I have a small multi-NIC Intel mini-PC running OPNsense on Proxmox really well, so hopefully it's just a misconfiguration somewhere. This is my config - I have 3 of the 4 NICs in pass-through for performance and not wanting anything else interfering with them, with the 4th one bridged for access to my LAN + a couple of VLANs:

Code: [Select]

agent: 1
args: -vnc 0.0.0.0:10
balloon: 0
bios: seabios
boot: order=scsi0
cores: 4
cpu: host
hostpci0: 0000:01:00,pcie=1
hostpci1: 0000:03:00,pcie=1
hostpci2: 0000:04:00,pcie=1
hotplug: disk,network,usb,cpu
machine: q35,viommu=virtio
memory: 8192
meta: creation-qemu=8.1.2,ctime=1701086589
name: BART
net0: virtio=BC:REDACTED:D5,bridge=vmbr0
numa: 1
onboot: 1
ostype: l26
scsi0: local-zfs:vm-100-disk-1,discard=on,iothread=1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=fa-REDACTED-88
sockets: 1
startup: order=2
tags: linux;vm
vmgenid: 84-REDACTED-bf

When I first installed this back in September 2023, the only real issues I had were that I could not get OPNsense to boot if I tried to use UEFI, hence why I have "seabios" for the BIOS, plus disabling ballooning or I would be stuck at 2GB RAM. Otherwise it's all pretty standard.


This is on PvE v8.2.4 and OPNsense v24.1 as I haven't been brave enough to upgrade to v24.7 yet.

[Vesalius]

Just so I am clear on your issue.

At the OPNsense level you want to bridge a passed-through X550-T2 port (LAN) Physically connected to a switch with a Proxmox virtual-only linux bridge using model virtio? So line speed to physical lan and Paravirtualized speed to Proxmox VM's through a single OPNsense bridge.

At the proxmox level, I assume this Linux bridge does not have an assigned Port/Slave (so not physically connected to anything)?

This OPNsense Bridge (passed-through X550-T2 port (LAN) + proxmox Net1) functions when the Proxmox Network device defined in the OPNsense VM is Model Intel E1000, but the Bridge fails if the Proxmox Network device defined in the OPNsense VM is Model Virtio?

AND this setup works correctly in PFSENSE Bridge (passed-through X550-T2 port (LAN) + proxmox Net1 Model Virtio)?

Have you tested to be sure that a proxmox Virtio network device in promox functions at expected speeds alone outside this bridge?

I would retry using a q35 based OPNsense VM. Latest Nonsubscription promox has moved to QEMU 9 as well and that may require a reboot to move the VM over.

[Crycaeon]

Perhaps you can post the VM config from Proxmox so we can see exactly how it's been configured?

A great idea, I'm not sure why I didn't do that in the first place. I will note that the hostpci0 device that I pass through is the base address of the networking card that I pass through for physical connections and is further enumerated, within OPNsense, into its 2 component interfaces that give me my physical WAN and LAN connections; I pass it through in this was so that, at the lowest level, I isolate it from Proxmox.

Code: [Select]

agent: 0
balloon: 2048
boot: order=scsi0;ide2
cores: 2
cpu: host
hostpci0: 0000:0e:00
ide2: none,media=cdrom
machine: q35
memory: 8192
meta: creation-qemu=8.1.5,ctime=1712721471
name: OPNsense
net0: e1000e=[Redacted],bridge=vmbr100
net1: virtio=[Redacted],bridge=vmbr100
numa: 0
onboot: 1
ostype: l26
parent: After24_7_9
scsi0: local-lvm:vm-100-disk-1,iothread=1,size=30G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=25c81be7-86b3-4923-8b40-3c4944d43b7a
sockets: 2
startup: order=1
tags: opnsense
vga: std
vmgenid: Redcated
I also was not able to get it working consistently with UEFI and had to stay with Seabios since sometimes the system failed to boot entirely. Not sure why that was but I have simply stuck with Seabios since.

This is on PvE v8.2.4 and OPNsense v24.1 as I haven't been brave enough to upgrade to v24.7 yet.

I have been more aggressive than normal when it comes to updating OPNsense since I have been hoping, for a while now, that an update might fix my issue. I tend to lean on the side of everyone who has said this is a Proxmox and not an OPNsense issue since I have used OPNsense with Virtio at work before but for the life of me I can't figure out why it would behave like this.

Just so I am clear on your issue.

Vesalius, you have characterized the situation perfectly in your breakdown.

Have you tested to be sure that a proxmox Virtio network device in promox functions at expected speeds alone outside this bridge?

Yes and even running by itself and activated the connection does not seem to correctly register. When looking at the interfaces menu I cannot see any settings for speed or duplex settings. However, running ifconfig shows that the interface is active and functioning at the correct speed but it is not reachable on its own by anything more than a ping.

If I do it right I will have attached pictures comparing the vtnet0 (Virtio) and em0 (E1000) interfaces from within OPNsense GUI and CLI.

[Taomyn]

Out of interest, how much RAM is OPNsense reporting as installed on the dashboard?


I can see you have a balloon set at 2048 with 8192 for the total, which is much like what I wanted, but that never worked for me and OPNsense reported only ever reported 2048 and would never go above this. I had to set balloon to 0, i.e. disable it, before it could see all 8GB. Again, not sure yet if this is something that 24.7 may have fixed with its newer OS - going to give the upgrade a go this coming weekend.

[Crycaeon]

Out of interest, how much RAM is OPNsense reporting as installed on the dashboard?

Dashboard shows 8155MB of memory. Not sure why the discrepancy between 8192 and that value. Also, balloon seems to have always worked for me.

[Crycaeon]

I'm gonna go ahead and mark this as closed since I have circumvented the issue by changing my setup. I ended up rerouting things to use my Proxmox management port as my LAN port for my network since externally I'm bottlenecked to gigabit anyway. Considering my management port isn't out of band I'm not really getting any great usage out of a dedicated management port so might as well do double duty there; a bit of extra CPU overhead won't hurt my system.

Not terribly happy I couldn't figure out what was going on but for now my VMs, local network, and WAN connection aren't bottlenecked so this works for my purposes.