OPNSense on Proxmox, 10Gb network awful throughput

[Nomsplease]

Bringing this thread over from Reddit, I am the OP just with a different user name, to hopefully get some more insight on what could be wrong here.

Reddit thread is here: https://www.reddit.com/r/opnsense/comments/17fjbbw/opnsense_on_proxmox_10gb_network_woes/

I have been having quite a lot of issues getting a virtualized OPNsense setup to pass 10Gb traffic anywhere near line speed. This is a fresh setup so I do not have existing vlans setup. I have been going at this for the past couple days without any success and I'm currently running on a mini PC to continue trying to find the solution to this issue.

Firstly the hardware the proxmox host is running on.

Board: Supermicro x11SSH-F

CPU: E3-1275V5

Ram: 64GB DDR4 UDIMM

Storage: SSDs in ZFS Mirror

Nic: X520-DA2


The VM has been setup on both virtual platforms, that being i440fx and Q35 with no change from either. It is setup to have 4 cores and 8 GB of ram. I have attempted to pass through the NIC as well as run it bridged through the host itself.

I have ran multi queue on the host in both 4 and 8, neither made any difference. I have tried numerous tunables to get the line speed with also no success. I have even gone as far as installing OPNSense on bare metal on this host, again this has not worked either.


Tunables I have tried:

Code: [Select]

hw.ibrs_disable=1
net.isr.maxthreads=-1
net.isr.bindthreads = 1
net.isr.dispatch = deferred
net.inet.rss.enabled = 1
net.inet.rss.bits = 6
kern.ipc.maxsockbuf = 614400000
net.inet.tcp.recvbuf_max=4194304
net.inet.tcp.recvspace=65536
net.inet.tcp.sendbuf_inc=65536
net.inet.tcp.sendbuf_max=4194304
net.inet.tcp.sendspace=65536
net.inet.tcp.soreceive_stream = 1
net.pf.source_nodes_hashsize = 1048576
net.inet.tcp.mssdflt=1240
net.inet.tcp.abc_l_var=52
net.inet.tcp.minmss = 536
kern.random.fortuna.minpoolsize=128
net.isr.defaultqlimit=2048
VM Setup:



Closest test to Line speed I have gotten, this was running all the tunables above and making all the vCPUs sockets instead of cores. This was able to reach 9Gb/s but only maintained it for 20 seconds and fell off hard.

Iperf when installed bare metal on the host hardware was only able to reach 1.1Gb/s


I even went as far as putting it on another host where my Truenas VM lives. This is pretty clearly telling me something in OPNsense is either misconfigured, or just outright is not working correctly with 10G hardware.

Anyone have any ideas that could maybe lead me in the right direction? I know plenty of other people have no issues running OPNsense in VMs on proxmox, I think the place im getting stuck on is the 10G side of things.

[Patrick M. Hausen]

How are your numbers with OPNsense on bare metal?

[Nomsplease]

How are your numbers with OPNsense on bare metal?

Worse or the same as virtualized, about 1.1Gb/s average.

[BruceOS]

1) try your tunable BUT

net.inet.rss.enabled = 0


- i had problems with that for updates under proxmox, maybe it is affecting throughput as well.


2) otherwise remove all tunables and go one by one (again)

3) AND I see you have 4 SOCKETS 1 CORE
AND I THINK maybe you meant 1 SOCKET 4 CORES with your E3-1275V5

SOCKET = CPU SOCKET - you have 1 E3-1275V5? Right
CORE= Number of cores available for the VM including your VirtIO Hardware. Maybe you are CPU throtelling ?

Even if you had 4 sockets, it would not be a good thing to multihread with different CPUs, maybe the is a use case i don't know out there


Client Settings for Proxmox

agent: 1
balloon: 0
boot: order=scsi0;ide2;net0
cores: 2
cpu: host,flags=+aes
efidisk0: local-lvm:vm-110-disk-0,efitype=4m,pre-enrolled-keys=1,size=16G
hostpci1: 0000:05:00,pcie=1,rombar=0   ##>> PCI pass through for WAN Interface
ide2: none,media=cdrom
machine: q35
memory: 3072
name: opnsnse
net0: virtio=99:99:YY:XX:XX:XX,bridge=vmbr0,queues=4   ##Standard queue is 1 parallel stream
net2: virtio=99:99:YY:XX:XX:XX,bridge=vmbr10,queues=4  ##Standard queue is 1 parallel stream
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-123-disk-0,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=xxyyzz #your personal (SeaBIOS)
sockets: 1
startup: order=1,up=1
vmgenid: xxyyzz #your personal


4) Try deactivating IPS / IDS for testing
Service -> Intrusion Detection _-> Settings
[ ] Enabled  #unchecked
[ ] IPS mode #unchecked

[Nomsplease]

1) try your tunable BUT

net.inet.rss.enabled = 0

I have actually been playing with the multi queue and rss options to see where this is all falling apart.

I have found that if I run the VM with 4 cores, multiqueue set to 4, rss.bits =2 and rss.enabled =1. I get 9.XGb/s for about 10 seconds then it just falls off back down to basically gigabit speeds where we started. It seems completely random if I actually get line speed or a nerfed speed out of this setup which seems entirely wrong.

Run with rss.enabled=0

Code: [Select]

root@OPNsense:~ # iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 172.16.1.90, port 50532
[  5] local 172.16.1.80 port 5201 connected to 172.16.1.90 port 50533
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   232 MBytes  1.94 Gbits/sec
[  5]   1.00-2.00   sec   172 MBytes  1.44 Gbits/sec
[  5]   2.00-3.00   sec   162 MBytes  1.36 Gbits/sec
[  5]   3.00-4.00   sec   154 MBytes  1.29 Gbits/sec
[  5]   4.00-5.00   sec   138 MBytes  1.16 Gbits/sec
[  5]   5.00-6.00   sec   138 MBytes  1.15 Gbits/sec
[  5]   6.00-7.00   sec   156 MBytes  1.30 Gbits/sec
[  5]   7.00-8.00   sec   127 MBytes  1.06 Gbits/sec
[  5]   8.00-9.00   sec   134 MBytes  1.13 Gbits/sec
[  5]   9.00-10.00  sec   165 MBytes  1.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.54 GBytes  1.32 Gbits/sec                  receiver
Run with Rss.enabled=1, rss.bits=2, multiqueue set to 4

Code: [Select]

-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
Accepted connection from 172.16.1.90, port 51523
[  5] local 172.16.1.80 port 5201 connected to 172.16.1.90 port 51524
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1000 MBytes  8.39 Gbits/sec
[  5]   1.00-2.00   sec  1.09 GBytes  9.41 Gbits/sec
[  5]   2.00-3.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec
[  5]   4.00-5.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   5.00-6.00   sec  1.09 GBytes  9.39 Gbits/sec
[  5]   6.00-7.00   sec  1.09 GBytes  9.39 Gbits/sec
[  5]   7.00-8.00   sec  1.10 GBytes  9.40 Gbits/sec
[  5]   8.00-9.00   sec  1.09 GBytes  9.41 Gbits/sec
[  5]   9.00-10.00  sec  1.09 GBytes  9.41 Gbits/sec
[  5]  10.00-10.00  sec  2.12 MBytes  8.80 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.8 GBytes  9.30 Gbits/sec                  receiver
And again right after the previous test, the router just fell on its face. Rss.enabled=1, rss.bits=2, multiqueue set to 4

Code: [Select]

-----------------------------------------------------------
Server listening on 5201 (test #3)
-----------------------------------------------------------
Accepted connection from 172.16.1.90, port 51553
[  5] local 172.16.1.80 port 5201 connected to 172.16.1.90 port 51554
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   1.00-2.00   sec   182 MBytes  1.53 Gbits/sec
[  5]   2.00-3.00   sec  73.8 MBytes   617 Mbits/sec
[  5]   3.00-4.00   sec   108 MBytes   910 Mbits/sec
[  5]   4.00-5.00   sec   151 MBytes  1.26 Gbits/sec
[  5]   5.00-6.00   sec   148 MBytes  1.24 Gbits/sec
[  5]   6.00-7.00   sec   153 MBytes  1.28 Gbits/sec
[  5]   7.00-8.00   sec   152 MBytes  1.28 Gbits/sec
[  5]   8.00-9.00   sec   171 MBytes  1.43 Gbits/sec
[  5]   9.00-10.00  sec   151 MBytes  1.27 Gbits/sec
[  5]   9.00-10.00  sec   151 MBytes  1.27 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.51 GBytes  1.30 Gbits/sec                  receiver
iperf3: the client has terminated

3) AND I see you have 4 SOCKETS 1 CORE
AND I THINK maybe you meant 1 SOCKET 4 CORES with your E3-1275V5

SOCKET = CPU SOCKET - you have 1 E3-1275V5? Right
CORE= Number of cores available for the VM including your VirtIO Hardware. Maybe you are CPU throtelling ?

Even if you had 4 sockets, it would not be a good thing to multihread with different CPUs, maybe the is a use case i don't know out there


Client Settings for Proxmox

agent: 1
balloon: 0
boot: order=scsi0;ide2;net0
cores: 2
cpu: host,flags=+aes
efidisk0: local-lvm:vm-110-disk-0,efitype=4m,pre-enrolled-keys=1,size=16G
hostpci1: 0000:05:00,pcie=1,rombar=0   ##>> PCI pass through for WAN Interface
ide2: none,media=cdrom
machine: q35
memory: 3072
name: opnsnse
net0: virtio=99:99:YY:XX:XX:XX,bridge=vmbr0,queues=4   ##Standard queue is 1 parallel stream
net2: virtio=99:99:YY:XX:XX:XX,bridge=vmbr10,queues=4  ##Standard queue is 1 parallel stream
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-123-disk-0,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=xxyyzz #your personal (SeaBIOS)
sockets: 1
startup: order=1,up=1
vmgenid: xxyyzz #your personal


4) Try deactivating IPS / IDS for testing
Service -> Intrusion Detection _-> Settings
[ ] Enabled  #unchecked
[ ] IPS mode #unchecked

The CPU thing was a test scenario, the vm is a 4 core 1 cpu machine again. No IDS or IPS running.

[Mars79]

Looking at your mentioned tunables I can see you already applied them from most likely
https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet
so it seems you already did some research 

Did you also take a look at the Performance tuning for IPS maximum performance thread on the Intrusion Detection and Prevention section of this forum?
Some useful information is being mentioned there as well. I personally have seen big improvements with the Flow Control tunable, but your mileage may vary of course.

https://forum.opnsense.org/index.php?topic=6590.0

[Nomsplease]

Looking at your mentioned tunables I can see you already applied them from most likely
https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet
so it seems you already did some research 

Did you also take a look at the Performance tuning for IPS maximum performance thread on the Intrusion Detection and Prevention section of this forum?
Some useful information is being mentioned there as well. I personally have seen big improvements with the Flow Control tunable, but your mileage may vary of course.

https://forum.opnsense.org/index.php?topic=6590.0

I read a lot your threadand plenty of others over the past few days trying to get this to work. I also do not use IDS or IPS, wouldnt even touch it until I could reliably get 10G speeds. From what I can tell this is a pretty common issue among users trying to get over 1 gig line speeds. Im surprised this is still such an issue in 2023 with how common 10g networks are and the increasing speed of ISPs. I have a friend getting 2.5G to his home shortly, and he is going to be in for a rude awakening when his firewall can no longer get to that internet speed.

Sucks really that this seems like such an issue, and it seems its related to BSD and its poor optimization for antyhing over 1G interfaces. All of my linux hosts have no issues being on 10G interfaces, even my MAC clients have no issues, so its just BSD that seems to have any issues with it. Im hoping someone else can come in with a solution, but seems I wont be running OPNSense anylonger..

[Monviech (Cedrik)]

Try a clean Freebsd 13 VM or a PFsense and look if you have the same performance numbers.

I dont think its FreeBSD, I mean Deciso sells 25GBit/s hardware appliances... I'm sure they can do what advertised.

https://shop.opnsense.com/dec4200-series-opnsense-enterprise-datacenter-rack-security-appliance/

[meyergru]

The dropping of speed points to something that is likely not software. You use X520-DA2, but with what SFP+ module attached to it? Optical, Ethernet or DAC?

Ethernet can get very hot... is there a drop on the interface speed? I had problems with Cat.5 cables when they were too long. Turned out to be the Cat.5 jacks, not the cabling itself.

[Nomsplease]

The dropping of speed points to something that is likely not software. You use X520-DA2, but with what SFP+ module attached to it? Optical, Ethernet or DAC?

Ethernet can get very hot... is there a drop on the interface speed? I had problems with Cat.5 cables when they were too long. Turned out to be the Cat.5 jacks, not the cabling itself.

This would make sense aside from the fact that it is not the case on a linux host. Anything other then BSD has no issues with these 10g interfaces, as shown in my screenshots against a truenas scale VM. The scale vm can maintain 10g line speeds for as long as I run the test. This is strictly a BSD issue.

I have ordered a few replacement nics to see if this is just an issue with the X520 interfaces, since this seems to be what everyone is having issues with. Its still recommended quite often, but maybe this recommendation is no longer a good one. I have had these interfaces for close to 10 years, so they are definitely moving on in their age.

[Nomsplease]

To rule out a BSD issue, I installed a FreeBSD 13 vm with the same setup as the OPNsense vm.

It does a lot better then OPNsense, but it also suffers from the speed dropping off randomly.

Code: [Select]

> iperf3 -c 172.16.1.221 -p 5201 -t 30
Connecting to host 172.16.1.221, port 5201
[  5] local 172.16.1.90 port 50331 connected to 172.16.1.221 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.06 GBytes  9.15 Gbits/sec
[  5]   1.00-2.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   2.00-3.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   3.00-4.00   sec  1.09 GBytes  9.41 Gbits/sec
[  5]   4.00-5.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   5.00-6.00   sec  1.09 GBytes  9.33 Gbits/sec
[  5]   6.00-7.00   sec  1.09 GBytes  9.39 Gbits/sec
[  5]   7.00-8.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   8.00-9.00   sec  1.10 GBytes  9.41 Gbits/sec
[  5]   9.00-10.00  sec  1.09 GBytes  9.40 Gbits/sec
[  5]  10.00-11.00  sec  1.09 GBytes  9.40 Gbits/sec
[  5]  11.00-12.00  sec  1.10 GBytes  9.41 Gbits/sec
[  5]  12.00-13.00  sec  1.10 GBytes  9.41 Gbits/sec
[  5]  13.00-14.00  sec  1.10 GBytes  9.41 Gbits/sec
[  5]  14.00-15.00  sec  1.09 GBytes  9.40 Gbits/sec
[  5]  15.00-16.00  sec  1.10 GBytes  9.41 Gbits/sec
[  5]  16.00-17.00  sec  1.10 GBytes  9.41 Gbits/sec
[  5]  17.00-18.00  sec  1.06 GBytes  9.12 Gbits/sec
[  5]  18.00-19.00  sec   302 MBytes  2.53 Gbits/sec
[  5]  19.00-20.00  sec   241 MBytes  2.02 Gbits/sec
[  5]  20.00-21.00  sec   397 MBytes  3.33 Gbits/sec
[  5]  21.00-22.00  sec   640 MBytes  5.37 Gbits/sec
[  5]  22.00-23.00  sec  1.10 GBytes  9.41 Gbits/sec
[  5]  23.00-24.00  sec  1.10 GBytes  9.41 Gbits/sec
[  5]  24.00-25.00  sec   530 MBytes  4.44 Gbits/sec
[  5]  25.00-26.00  sec   379 MBytes  3.18 Gbits/sec
[  5]  26.00-27.00  sec  1.02 GBytes  8.72 Gbits/sec
[  5]  27.00-28.00  sec  1.10 GBytes  9.41 Gbits/sec
[  5]  28.00-29.00  sec  1.10 GBytes  9.41 Gbits/sec
[  5]  29.00-30.00  sec  1.08 GBytes  9.30 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.00  sec  28.5 GBytes  8.17 Gbits/sec                  sender
[  5]   0.00-30.00  sec  28.5 GBytes  8.17 Gbits/sec                  receiver

iperf Done.
Compare this to a Debian based VM (Truenas Scale), its speed never dips below 9Gb/s.

Code: [Select]

> iperf3 -c truenas -p 5201 -t 30
Connecting to host truenas, port 5201
[  7] local 172.16.1.90 port 50664 connected to 172.16.1.10 port 5201
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec
[  7]   1.00-2.00   sec  1.09 GBytes  9.37 Gbits/sec
[  7]   2.00-3.00   sec  1.09 GBytes  9.41 Gbits/sec
[  7]   3.00-4.00   sec  1.08 GBytes  9.30 Gbits/sec
[  7]   4.00-5.00   sec  1.08 GBytes  9.26 Gbits/sec
[  7]   5.00-6.00   sec  1.09 GBytes  9.39 Gbits/sec
[  7]   6.00-7.00   sec  1.10 GBytes  9.41 Gbits/sec
[  7]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec
[  7]   8.00-9.00   sec  1.09 GBytes  9.36 Gbits/sec
[  7]   9.00-10.00  sec  1.10 GBytes  9.41 Gbits/sec
[  7]  10.00-11.00  sec  1.09 GBytes  9.37 Gbits/sec
[  7]  11.00-12.00  sec  1.09 GBytes  9.38 Gbits/sec
[  7]  12.00-13.00  sec  1.10 GBytes  9.41 Gbits/sec
[  7]  13.00-14.00  sec  1.10 GBytes  9.41 Gbits/sec
[  7]  14.00-15.00  sec  1.09 GBytes  9.39 Gbits/sec
[  7]  15.00-16.00  sec  1.08 GBytes  9.28 Gbits/sec
[  7]  16.00-17.00  sec  1.10 GBytes  9.41 Gbits/sec
[  7]  17.00-18.00  sec  1.09 GBytes  9.41 Gbits/sec
[  7]  18.00-19.00  sec  1.09 GBytes  9.38 Gbits/sec
[  7]  19.00-20.00  sec  1.09 GBytes  9.35 Gbits/sec
[  7]  20.00-21.00  sec  1.10 GBytes  9.41 Gbits/sec
[  7]  21.00-22.00  sec  1.09 GBytes  9.38 Gbits/sec
[  7]  22.00-23.00  sec  1.10 GBytes  9.41 Gbits/sec
[  7]  23.00-24.00  sec  1.09 GBytes  9.40 Gbits/sec
[  7]  24.00-25.00  sec  1.10 GBytes  9.42 Gbits/sec
[  7]  25.00-26.00  sec  1.10 GBytes  9.41 Gbits/sec
[  7]  26.00-27.00  sec  1.10 GBytes  9.41 Gbits/sec
[  7]  27.00-28.00  sec  1.09 GBytes  9.39 Gbits/sec
[  7]  28.00-29.00  sec  1.10 GBytes  9.41 Gbits/sec
[  7]  29.00-30.00  sec  1.10 GBytes  9.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-30.00  sec  32.8 GBytes  9.38 Gbits/sec                  sender
[  7]   0.00-30.00  sec  32.8 GBytes  9.38 Gbits/sec                  receiver

iperf Done.
There is an issue with BSD here, either with the X520 interface, or something else within the OS. I will test the other interface cards when they arrive to see if they can improve the stability. There is still an issue with OPNsense here though where it is 20% the speed out of the box with the 10g interfaces then its FreeBSD base OS.

[diokobol]

I take the liberty of following up on this topic by pointing out that I have been living with this problem for months... several tests carried out both in proxmox, vmware and baremetal environments on HPE Proliand dl380 gen9 with dual E5-2697v4 processors and 2 Intel X540 dual BaseT network cards without getting 4/5(6 Gbps) results. IPS disabled. With freebsd 13 installed as it from DVD 9/9.5 even if not constant. Linux Debian 11 and 12 seems to present no problems both in a virtualized environment and in a barmetal environment. I also find it really strange that no one from TeamSviluppo has encountered similar problems.

I will follow up thanks for your attention.

[Cipher]

I am in the same situation as you guys, running the software on a bare-metal HPE server, the speed is limited to 1Gbps. However, when I run opnsense on VMware, I achieve a much better speed of 9.1Gbps.

[Kinerg]

There is an issue with BSD here, either with the X520 interface, or something else within the OS. I will test the other interface cards when they arrive to see if they can improve the stability. There is still an issue with OPNsense here though where it is 20% the speed out of the box with the 10g interfaces then its FreeBSD base OS.

Hi, have you managed to test the new adapters?

[tillsense]

Hi all,

would you like to test 14? i'm very interested to see how it looks here!

https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-disc1.iso

cheers
till

ps.: ..and with the new default CPU from pve8 > x86-64-v2-AES