Messages

What I do not get is why turning off hardware offloading (which is the default) does break routing. I would expect that maybe your gateway settings are off, given that you do not use OpnSense as your main gateway and that the WAN on OpnSense is disabled.

If that was the case, you could well ping any address on either side of OpnSense, yet a machine from one side will not ping any machine on the other because its default gateway is not OpnSense, but your main router.

Crap, did I write it wrong? Sorry if it wasn't clear. Enabling hardware offloading, specifically checksum offloading, breaks routing for me (though ICMP ping goes through). Once I disable checksum offloading, routing works again. Oddly enough, traceroutes also fail when csum offload is enabled.
FWIW, I know that hw.vtnet.csum_disable=1 (and reboot) is basically the same as ifconfig vtnetX -txcsum -rxcsum

I have double checked my gateways just now, and also have static routes added.

lan1 vm:

Code Select Expand

default via 10.4.2.1 dev enp6s18 proto dhcp src 10.4.2.184 metric 100
10.4.2.0/24 dev enp6s18 proto kernel scope link src 10.4.2.184 metric 100
10.4.3.0/24 via 10.4.2.2 dev enp6s18
lan2 vm:

Code Select Expand

default via 10.4.3.2 dev enp6s18 proto dhcp src 10.4.3.101 metric 100
10.4.3.0/24 dev enp6s18 proto kernel scope link src 10.4.3.101 metric 100

FWIW, your tagline/signature looks like a similar fanless box as what I'm running. You're right. My opnsense box right now is purely being used as a router, forwarding packets between lan1 and lan2. I tried disabling the opnsense firewall too, but that also had no effect on iperf3 performance between vms on either side.

I just stood up a openwrt router, and out of the box I'm seeing around 9.3Gbps iperf3 between lan1 and lan2. I'm turning it off though, openwrt really looks horrible and the GUI gives me a headache.

I do not see this, there must be something else wrong with that. What are lan1 and lan2? Do they have separate subnets or are they bridged? If the latter, did you set all the neccessary tuneables?

lan1=10.4.2.0/24
lan2=10.4.3.0/24
opnsense vm sits between on 10.4.2.2 and 10.4.3.2
i have a live edgerouter serving my internet at 10.4.2.1

my proxmox host nics -> linux bridges -> opnsense virtio
enp1s0->vmbr0->wan(disabled)
enp2s0->vmbr1->lan1
enp3s0->vmbr2->lan2
enp4s0->vmbr3->unused

The ubuntu vms I move between lan1 and lan2 for iperf3 testing.

I've set a boatload of tuneables as per what I read from here: https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/

Summarized:

Code Select Expand

hw.ibrs_disable=1
net.isr.maxthreads=-1
net.isr.bindthreads=1
net.isr.dispatch=deferred
net.inet.rss.enabled=1
net.inet.rss.bits=1
kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=4194304
net.inet.tcp.recvspace=65536
net.inet.tcp.sendbuf_inc=65536
net.inet.tcp.sendbuf_max=4194304
net.inet.tcp.sendspace=65536
net.inet.tcp.soreceive_stream=1
net.inet.tcp.mssdflt=1240
net.inet.tcp.abc_l_var=52
net.inet.tcp.minmss=536
kern.random.fortuna.minpoolsize=128
net.isr.defaultqlimit=2048
I've also got all the hardware offload and filtering disabled. This is my first foray into proxmox and opnsense, so I hope I'm not doing anything obviously wrong? It is quite possible though!

One thing I haven't yet done is reset the BIOS of my N100 proxmox server.

Apologies for the reponse lag.

You guys were right, as well as basically everything else I've read about vtnet offloading. With the checksum offload enabled, I could get some decent iperf3 numbers directly to/from my opnsense vm, but that was basically a moot point, since this it couldn't route! It is very strange, since I could ping through the router from lan1 to lan2 via opnsense, but I couldn't tcp connect between the lans. I've never seen anything like it.

Once I disabled checksum offload (I forget if it was tx or rx) routing was working, though I'm still surprised I'm seeing less than 4Gbps between vms on the same proxmox host when traversing opnsense. I guess I can try a different router to compare.

FWIW, setting hw.vtnet.csum_disable=1 in a FreeBSD 14.2 vm also resulted in LRO and TSO to become permanently disabled as well, so this seems to be more of a freebsd kernel bug than an opnsense one.

Running OPNsense 25.1(.1) on Proxmox 8.3.3 on an N100 box with 4x i226-v NICs, and I've been running into an iperf3 performance issue. I'm using virtio for all the VM networking, no pass-through.

With Ubuntu and FreeBSD VMs, I would see around 15-18Gbps between them, and around 20-25Gbps with Proxmox. Even with OPNsense 24.7, I saw similar numbers. But with OPNsense 25.1, I would hit a 5Gbps ceiling no matter to VMs or Proxmox. After many days of testing, I found that OPNsense 25.1 by default disables checksum offloading AND disables the GUI offload control. I've been banging my head why I couldn't toggle the RXCSUM/TXCSUM/LRO/TSO options via ifconfig like I could with OPNsense 24.7. I've found that when hw.vtnet.csum_disable=1, even things like TSO and LRO cannot be toggled on. The options ALWAYS look like this:

Code Select Expand

vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN (lan)
        options=90028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether bc:24:11:55:6e:37
        inet 10.4.2.2 netmask 0xffffff00 broadcast 10.4.2.255
        inet6 fe80::be24:11ff:fe55:6e37%vtnet0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Regardless if you try to do things like 'ifconfig vtnet0 lro' or 'ifconfig vtnet tso4', they wouldn't do anything.

Are LRO/TSO supposed to be affected by this tunable? Whatever the case is, setting hw.vtnet.csum_disable=0 then enabling the 3 offload settings in the GUI then rebooting, has restored the iperf3 performance in OPNsense 25.1 (for me).

NB: The manpage for vtnet seems to suggest that hw.vtnet.csum_disable SHOULD NOT have any affect on LRO or TSO4.
https://man.freebsd.org/cgi/man.cgi?vtnet(4)