Intermittant networking between resets with opnsense vm and pci nic pass-through

[dieterarn]

Hi all,

I've read that nic pass-though is the most secure way to virtualize a router/firewall. I'm using proxmox as my host with allows me to do a bunch of things since i can reuse the underlying hardware for serveral other VMs. I followed this guide on the great proxmox wiki and was able to pass-thought a 4 port intel nic to a vm running opnsense (dell v pro 1000). It all works - in fact I'm typing up this post "through" the firewall right now! Currently the vm is 17.1.2 but i had the same problem with 17.1.

As an aside i was able to solve the opnsense on zfs problem https://forum.opnsense.org/index.php?topic=3602.msg12620#msg12620 that i posted about previously in a round-about sort of way since proxmox supports booting using the zfs on linux stack.

The problem is this: When i reboot the opnsense vm networking randomly doesn't work. The boot sequence shows the machine finding the NICs and setting there state to "up" but sometimes the lan interface is un-pingable and the wan interface doesn't receive an ip though dhcp from my isp.

Could this be a host problem or a guest problem? I'm not even sure where to start googleing on this problem... i've cross post this to the proxmox forum in-case it is a host problem. Does anybody out their have any guest side ideas about what could be causing this?

[Nnyan]

I was thinking of passing through my quad port to OPNsense but I had so many other things to worry about I just didn't on this pass.  I know for ESXi the VMWare drivers for the network card "seems" to get installed and "sees" the card but it just will not work in OPNSense or pfSense.  Once I set it to the Intel E1000 driver they work fine.  So I would ask you to check your driver.

[dieterarn]

Hi Nnyan,

Do you mean check the driver on the host/hypervisor side (proxmox) or the guest side (opnsense)? It didn't occur to me that the host would bother loading drivers if using pci pass-through. I though host side drivers were not involved at all once the card was passed through.

on the host side get:

Code: [Select]

root@proxmox1:/rpool/data# lspci | grep Network
06:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
06:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
07:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
07:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
09:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
09:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

and

Code: [Select]

root@proxmox1:/rpool/data# dmesg | grep Ethernet
[    3.400241] Intel(R) Gigabit Ethernet Network Driver - version 5.3.5.3
[    3.615780] igb 0000:06:00.0: Intel(R) Gigabit Ethernet Network Connection
[    3.827716] igb 0000:06:00.1: Intel(R) Gigabit Ethernet Network Connection
[    4.047801] igb 0000:07:00.0: Intel(R) Gigabit Ethernet Network Connection
[    4.263806] igb 0000:07:00.1: Intel(R) Gigabit Ethernet Network Connection
[    4.458458] igb 0000:09:00.0: Intel(R) Gigabit Ethernet Network Connection
[    4.648021] igb 0000:09:00.1: Intel(R) Gigabit Ethernet Network Connection

pci device 6 & 7 are the quad port card and are currently passed through...

what the guest side (opnsense) has to say about this ...

Code: [Select]

root@OPNsense:~ # pciconf -lv
...
igb0@pci0:0:16:0: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82575GB Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb1@pci0:0:16:1: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82575GB Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb2@pci0:0:17:0: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82575GB Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb3@pci0:0:17:1: class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82575GB Gigabit Network Connection'
    class      = network
    subclass   = ethernet
...

but weirdly dmesg entrys seem to be in triplicate....

Code: [Select]

root@OPNsense:~ # dmesg | grep Ethernet
igb0: Ethernet address: 00:1b:21:2a:4f:e8
igb1: Ethernet address: 00:1b:21:2a:4f:e9
igb2: Ethernet address: 00:1b:21:2a:4f:ec
igb3: Ethernet address: 00:1b:21:2a:4f:ed
em0: Ethernet address: c6:48:af:7a:41:68
igb0: Ethernet address: 00:1b:21:2a:4f:e8
igb1: Ethernet address: 00:1b:21:2a:4f:e9
igb2: Ethernet address: 00:1b:21:2a:4f:ec
igb3: Ethernet address: 00:1b:21:2a:4f:ed
em0: Ethernet address: c6:48:af:7a:41:68
igb0: Ethernet address: 00:1b:21:2a:4f:e8
igb1: Ethernet address: 00:1b:21:2a:4f:e9
igb2: Ethernet address: 00:1b:21:2a:4f:ec
igb3: Ethernet address: 00:1b:21:2a:4f:ed
em0: Ethernet address: c6:48:af:7a:41:68
root@OPNsense:~ # dmesg | grep Network
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe080-0xe09f mem 0xfea40000-0xfea5ffff,0xfde00000-0xfdffffff,0xfeaf0000-0xfeaf3fff irq 11 at device 16.0 on pci0
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0a0-0xe0bf mem 0xfea60000-0xfea7ffff,0xfe000000-0xfe1fffff,0xfeaf4000-0xfeaf7fff irq 10 at device 16.1 on pci0
igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0c0-0xe0df mem 0xfea80000-0xfea9ffff,0xfe200000-0xfe3fffff,0xfeaf8000-0xfeafbfff irq 10 at device 17.0 on pci0
igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0e0-0xe0ff mem 0xfeaa0000-0xfeabffff,0xfe400000-0xfe5fffff,0xfeafc000-0xfeafffff irq 10 at device 17.1 on pci0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0xe000-0xe03f mem 0xfeac0000-0xfeadffff irq 10 at device 18.0 on pci0
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe080-0xe09f mem 0xfea40000-0xfea5ffff,0xfde00000-0xfdffffff,0xfeaf0000-0xfeaf3fff irq 11 at device 16.0 on pci0
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0a0-0xe0bf mem 0xfea60000-0xfea7ffff,0xfe000000-0xfe1fffff,0xfeaf4000-0xfeaf7fff irq 10 at device 16.1 on pci0
igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0c0-0xe0df mem 0xfea80000-0xfea9ffff,0xfe200000-0xfe3fffff,0xfeaf8000-0xfeafbfff irq 10 at device 17.0 on pci0
igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0e0-0xe0ff mem 0xfeaa0000-0xfeabffff,0xfe400000-0xfe5fffff,0xfeafc000-0xfeafffff irq 10 at device 17.1 on pci0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0xe000-0xe03f mem 0xfeac0000-0xfeadffff irq 10 at device 18.0 on pci0
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe080-0xe09f mem 0xfea40000-0xfea5ffff,0xfde00000-0xfdffffff,0xfeaf0000-0xfeaf3fff irq 11 at device 16.0 on pci0
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0a0-0xe0bf mem 0xfea60000-0xfea7ffff,0xfe000000-0xfe1fffff,0xfeaf4000-0xfeaf7fff irq 10 at device 16.1 on pci0
igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0c0-0xe0df mem 0xfea80000-0xfea9ffff,0xfe200000-0xfe3fffff,0xfeaf8000-0xfeafbfff irq 10 at device 17.0 on pci0
igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0e0-0xe0ff mem 0xfeaa0000-0xfeabffff,0xfe400000-0xfe5fffff,0xfeafc000-0xfeafffff irq 10 at device 17.1 on pci0
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0xe000-0xe03f mem 0xfeac0000-0xfeadffff irq 10 at device 18.0 on pci0
root@OPNsense:~ #