26.1.6 only works with E1000 using Proxmox

[Opnsensing-some-issues]

Hello,

I'm running a few i40e NICs on a Supermicro chassis for basic routing and SNAT. I haven't even gotten that far unfortunately though. Basically, with `qm set 100 --net0 virtio,bridge=vmbr0` I see slowness, partial loading, and inevitable unresponsiveness to the MGMT portal and therefore can't proceed. If I change `net0 e1000` everything works great without changing anything in `/etc/network/interfaces` or anywhere else. I have hardware offloading disabled (all 3). I tried disabling hardware offloading within Proxmox as well via `ethtool -k gro off`... and the rest of that command. Not sure what to do as without stable LAN over a bridge, this project is dead in the water with nothing I know that can easily replace it. i40e NICs are 10G so can't deal with e1000. I appreciate your help

[meyergru]

Probably this helps: https://forum.opnsense.org/index.php?topic=44159.0

[Opnsensing-some-issues]

Hmm. I looked through the doc, tried some tunables, but am not seeing network configuration I believe I should try. It looks like I have tried everything in that doc now, except for NATing between Proxmox and OPNSense which I don't believe I need or want.

[Opnsensing-some-issues]

Strangely, only the web UI seems impacted at this point. I can ssh fine. I tried large packet ping tests with DF bit set or off. Everything passes. So strange is almost like a TCP fragmentation issue but not

[meyergru]

I suppose there are no FreeBSD drivers for i40e, so you can only use virtio networking. The virtio drivers are pretty good in that there is no additional overhead to emulate any "real" hardware, so there should be no slowness, provided you do everything that is mentioned in the guide.

What I do not get: Even if virtio emulation does not work as expected and E1000 does, this is but a matter of performance, not functionality per se. So what is the problem with using E1000? AFAIK, there is no specific limit with either virtio nor E1000, modulo that any emulation layer through virtualisation limits performance with speeds way higher than 1 Gbps.

Without any specific details, it is hard to say more. Partial loading and general network slowness can be caused by many things that could also be occur if Proxmox was not involved, like wrong MTU settings (did you try to use jumbo packets?). By using OpnSense under Proxmox, you certainly make the setup more complex, that's for sure.

[Opnsensing-some-issues]

Isn't E1000 a 1Gbps interface? I thought that was the hard coded limit in the module hence why I'm shying away. I switched to vmxnet3 and it's working again. I suppose I just need this to spin up a basic config then will be able to switch to SR-IOV. One thing is sure: I have tried ever possible knob for virtio and it does not work. Here was the ping output:

C:\Users\bruh>ping 192.168.1.1 -l 1472 -f

Pinging 192.168.1.1 with 1472 bytes of data:
Reply from 192.168.1.1: bytes=1472 time=1ms TTL=64

Ping statistics for 192.168.1.1:
    Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 1ms, Average = 1ms
Control-C
^C
C:\Users\bruh>ping 192.168.1.1 -l 1473 -f

Pinging 192.168.1.1 with 1473 bytes of data:
Packet needs to be fragmented but DF set.

Ping statistics for 192.168.1.1:
    Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),
Control-C
^C
C:\Users\bruh>ping 192.168.1.1 -l 4000

Pinging 192.168.1.1 with 4000 bytes of data:
Reply from 192.168.1.1: bytes=4000 time=1ms TTL=64
Reply from 192.168.1.1: bytes=4000 time=1ms TTL=64

Ping statistics for 192.168.1.1:
    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 1ms, Average = 1ms

[meyergru]

The physical interface is limited to 1 Gbps, I doubt that the driver itself does anything w/r to timing. However, virtio on the vistualisation border should work just fine. IDK how SR-IOV comes into play here, because you do not / should not pass thru the NICs.

Just to make this clear: There are two basic ways you can do this:

1. By passing thru the physical NIC hardware to the VM guest. This absolutely needs a working FreeBSD for the hardware. I do not recommend it. This is where technologies like SR-IOV comes into play.

2. By attaching a virtual network card to the PVE host bridge. In the VM, you can then select which emulated hardware you want to present to the guest, so either E1000 or virtio or whatever hardware your guest supports. For OpnSense, I recommend this way and also, using the virtio drivers.
Virtio would show up as virtioX as network device names under OpnSense.

You MTU looks O.K., so this should work fine. Maybe the physical NIC has some optimisations to coalesce interrupts. This is often the case for high-speed NICs in order to handle traffic more efficiently. It may well be that this interferes and makes low-volume traffic look "choppy".

I would try to use another NIC type to rule out a hardware/driver problem on the PVE side of things, especially, because there are many problem reports for those adapters over on the Proxmox forum: https://forum.proxmox.com/search/19853344/?q=I40e+nic