I've been through most of the speed threads here and think I've tried everything mentioned, but having switched to OPNsense from Untangle, I'm getting marginally slower downloads on otherwise identical VMs.
My hardware is old, but relatively common so I don't think it has any major quirks to work around. I'm running proxmox on a Qotom PC with 4 intel 1G nics.
VMs are identical with virtio bridges, and the i5 CPU passed as 'host'. It's been pretty solid for about 3-4 years on Untangle, but the licensing has changed and I'm shopping alternatives now.
I'm running speed tests on a LAN machine. My ISP service is 1G down / 40M up.
I believe all hardware offload is disabled. My OPNsense install is fresh and nearly default settings.
During tests, cpu report in the OPNsense dash never report peaks higher than 50% and it mostly just runs under 10%. Proxmox reports a bit higher, but nothing I see that's concerning.
With untangle I get:
- directly on VM: untested
- LAN client: 850-900 down
With OPNsense:
- directly on VM: 920 down
- LAN client /OPNsense no add ins: 630-690 down
- LAN client w/ wireguard enabled: 530 down
- LAN client w/ wireguard and ips: 430 down
- LAN client w/ zenarmor: 380 down --edit, just added
It seems the LAN interface is the bottleneck, but I'm just a hobbyist so don't want to jump to conclusions. So far the only change I've tried is setting "generic-receive-offload off" in proxmox, but I'm not seeing much change.
I can live with some performance hit on the bridges, but I plan to add more services like zenarmor and am concerned that it will get more sluggish.
Any suggestions for things I've missed?
A bit more tinkering. I ended up passing through 2 of the interfaces, which was a bit of an adventure, but ultimately ended up with the same performance.
Switching back to untangle, I noticed it has slowed down and traced it to the interfaces falling back to half duplex. I corrected that and untangle was back up to speed. That got me looking at opnsense interface settings.
It looks like the LAN is properly autodetecting 1G full. Changing the setting to 1000baseT-full makes no difference.
On the WAN side I see there is no place to set it in the GUI. OPNsense has it detected as:
- Media 10Gbase-T <full-duplex>
- Media (Raw) Ethernet autoselect (10Gbase-T <full-duplex>)
But it's only a 1G Intel interface. Any idea if this could be causing trouble, or is it okay that it's over-spec'd?
QuoteWith OPNsense:
directly on VM: 920 down
LAN client /OPNsense no add ins: 630-690 down
LAN client w/ wireguard enabled: 530 down
LAN client w/ wireguard and ips: 430 down
LAN client w/ zenarmor: 380 down --edit, just added
So you tested between a HOST and OPNSense directly? If yes then
A. LAN client /OPNsense no add ins: 630-690 down -
iperf should be thru OPNsense not towards, meaning host to hostB. LAN client w/ wireguard enabled: 530 down -
this is CPU heavy it can limit you to such speed, but after changing the base to FreeBSD 14.1 WG should be improved there is a topic abotu this try to look it up. Did you also configure the Normalization as is in the guide?C.LAN client w/ wireguard and ips: 430 down
D.LAN client w/ zenarmor: 380 down --edit, just added
I am not sure about Suricata, but ZA limits you to only 1 CORE usage as it does not have a multi-thread support. Thus is heavy dependent on the performance and clock speed of a single CPU. Having WG + ZA + depending what else you run on OPN, can result in lower throughput.Did you try to enable RSS? I have seen improvement in performance when using ZA.
Regards,
S.
> But it's only a 1G Intel interface. Any idea if this could be causing trouble, or is it okay that it's over-spec'd?
It is what happens when the interface is virtualised and not a problem in itself. That is the speed of virtio between VM and host only as it is I believe (not seen the code) coded that way.
Quote from: cookiemonster on January 06, 2025, 11:12:25 AM> But it's only a 1G Intel interface. Any idea if this could be causing trouble, or is it okay that it's over-spec'd?
It is what happens when the interface is virtualised and not a problem in itself. That is the speed of virtio between VM and host only as it is I believe (not seen the code) coded that way.
Jop, virtio interfaces are negotiated at 10G speeds, because that the speed the Proxmox bridge communicates with its VMs. This is well expected. Back in the past it showed 100G :D, but they actually fixed it to 10G.
https://www.reddit.com/r/Proxmox/comments/lhe3vv/proxmox_show_10gb_setting_for_1gb_nic_adapter/
Regards,
S.
Have you tried iperf3 to doublecheck the link between LAN client and LAN interface at OPNsense VM ?
# @OPNsense, pkg install iperf3
iperf3 -s -B <lan-ip-opnsense>
# @LAN-client
iperf3 -c <lan-ip-opnsense> -P2
iperf3 -c <lan-ip-opnsense> -P2 -R
should saturate the link almost full gigabit
just finished my re-tests of OPN running on Proxmox:
1. setup OPNsense VM network adapters with multiqueue https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_network_device (e.g. set 4, if vCPU == 4, as in my case)
2. setup net.isr bind-/max-threads via System-Settings-Tunables https://docs.opnsense.org/troubleshooting/performance.html#kernel-support
I use two additional VMs (in WAN, and LAN) and run iperf3 server/client on them respectively, OPNsense VM is doing default NAT between them. Proxmox connects all via two Linux vmbr (WAN, LAN). Another option would be to run iperf3 server or client on Proxmox host itself.
My result is 2 Gigabit/s in either direction for "iperf3 -P2 -t60" and 4-vCPU OPNsense VM.
Let us know your LAN client performance result, pls.
T.
I got sidetracked but finally looping back. I was able to improve performance with both interfaces passed through, and I'm getting 960+ down both OPNsense (out of the box config) and untangle.
Unfortunately too long had passed and I forgot where I was, so I just started over from scratch. I suspect the problem was an incorrect passthrough setup.