Hello,
My hypervisor has 2 nics:
- 1x 1Gb/s
- 1x 25Gb/s connected to a switch ( which lead to my upstream ).
I use OPNsense 24.7.12_2-amd64 in a XCP-NG virtual machine which has 4c / 8Gb ram / 50Gb nvme.
The CPU of the server is AMD EPYC 4464P (3,7 GHz )
i'm in Hardware virtualization with paravirtualization drivers enabled (PVHVM) with Realtek 8139
i attached 2x the 25gb/s link ( 1 for wan, 1 for the local VLAN ).
I can't get better than 3Gb/s
[18:39 server-1 ~]# iperf3 -c 10.255.0.254
Connecting to host 10.255.0.254, port 5201
[ 4] local 10.255.1.3 port 38188 connected to 10.255.0.254 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 351 MBytes 2.95 Gbits/sec 362 536 KBytes
[ 4] 1.00-2.00 sec 331 MBytes 2.78 Gbits/sec 4 582 KBytes
[ 4] 2.00-3.00 sec 354 MBytes 2.97 Gbits/sec 21 465 KBytes
[ 4] 3.00-4.00 sec 341 MBytes 2.86 Gbits/sec 5 556 KBytes
[ 4] 4.00-5.00 sec 310 MBytes 2.60 Gbits/sec 15 441 KBytes
[ 4] 5.00-6.00 sec 354 MBytes 2.97 Gbits/sec 64 644 KBytes
[ 4] 6.00-7.00 sec 345 MBytes 2.89 Gbits/sec 6 530 KBytes
[ 4] 7.00-8.00 sec 312 MBytes 2.62 Gbits/sec 7 671 KBytes
[ 4] 8.00-9.00 sec 348 MBytes 2.92 Gbits/sec 14 581 KBytes
[ 4] 9.00-10.00 sec 326 MBytes 2.74 Gbits/sec 50 571 KBytes
When i run the iperf, the cpu is full !
When i disable the firewall in settings, the iperf is 2x more performant.
[18:45 server-1 ~]# iperf3 -c 10.255.0.254 -t 10000
Connecting to host 10.255.0.254, port 5201
[ 4] local 10.255.1.3 port 38318 connected to 10.255.0.254 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 617 MBytes 5.17 Gbits/sec 464 742 KBytes
[ 4] 1.00-2.00 sec 540 MBytes 4.53 Gbits/sec 57 671 KBytes
[ 4] 2.00-3.00 sec 639 MBytes 5.36 Gbits/sec 118 522 KBytes
[ 4] 3.00-4.00 sec 646 MBytes 5.42 Gbits/sec 58 636 KBytes
[ 4] 4.00-5.00 sec 665 MBytes 5.58 Gbits/sec 35 599 KBytes
[ 4] 5.00-6.00 sec 699 MBytes 5.86 Gbits/sec 150 698 KBytes
[ 4] 6.00-7.00 sec 616 MBytes 5.16 Gbits/sec 128 702 KBytes
[ 4] 7.00-8.00 sec 692 MBytes 5.82 Gbits/sec 156 735 KBytes
^C[ 4] 8.00-8.53 sec 380 MBytes 6.03 Gbits/sec 50 509 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-8.53 sec 5.37 GBytes 5.40 Gbits/sec 1216 sender
[ 4] 0.00-8.53 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated
I precise that i also tried to remove ALL MY RULES, i don't see any difference in term of cpu usage.
Why do i use all theses CPU ?
And the final question, yesterday, after a ton of test ( which i don't note somewhere of course ) i figured to have my 17GB/s !!! , but after a reboot, my bandwith come back to 3GB/s...
[19:52 server-1 ~]# iperf3 -c 10.255.1.254
Connecting to host 10.255.1.254, port 5201
[ 4] local 10.255.1.3 port 59300 connected to 10.255.1.254 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.80 GBytes 15.4 Gbits/sec 265 682 KBytes
[ 4] 1.00-2.00 sec 1.79 GBytes 15.3 Gbits/sec 245 743 KBytes
[ 4] 2.00-3.00 sec 1.88 GBytes 16.2 Gbits/sec 216 1014 KBytes
[ 4] 3.00-4.00 sec 1.91 GBytes 16.4 Gbits/sec 138 1.60 MBytes
[ 4] 4.00-5.00 sec 1.85 GBytes 15.9 Gbits/sec 153 1.98 MBytes
[ 4] 5.00-6.00 sec 2.00 GBytes 17.2 Gbits/sec 262 638 KBytes
[ 4] 6.00-7.00 sec 1.92 GBytes 16.5 Gbits/sec 351 944 KBytes
[ 4] 7.00-8.00 sec 1.78 GBytes 15.3 Gbits/sec 241 2.06 MBytes
[ 4] 8.00-9.00 sec 1.97 GBytes 16.9 Gbits/sec 240 655 KBytes
[ 4] 9.00-10.00 sec 1.58 GBytes 13.6 Gbits/sec 210 2.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 18.5 GBytes 15.9 Gbits/sec 2321 sender
[ 4] 0.00-10.00 sec 18.5 GBytes 15.9 Gbits/sec receiver
When i run no trafic, my CPU usage is at 13%-20%.
Thanks !
I run OPNsense (4 cores, 8GB RAM, 64GB disk) virtualized on Proxmox on a N305 fanless NUC.
Looking at top at the console, the system is largely idle (sometimes completely) under background use:
last pid: 30707; load averages: 0.22, 0.23, 0.21 up 9+23:37:29 15:00:58
90 processes: 1 running, 89 sleeping
CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Under 300Mbps load (my ISP max), interrupts go up to ~5%.
Proxmox is pretty lightweight and has reasonable overhead (the above seems to translate to a mix of user/system ~8%).
I have no interest in pushing higher load (would force me to run some inter VLAN arbitrary test).
I'd check where the CPU is going. top is a start.
Do you actually need that kind of bandwidth (LAN/WAN or Inter-VLAN) or is it merely a test?
Oh, I forgot to mention it, your last test shows this:
[ 4] local 10.255.1.3 port 59300 connected to 10.255.1.254 port 5201
where other tests had 10.255.0.254 for destination.
If that last test was within VLAN, and the others were not (i.e. inter-VLAN), that could explain the difference.
The VLAN traffic may not touch the router at all.
OPN is running off a single NIC with one VLAN for LAN and another for WAN?
I've never used that config. It's on my to-do...
Hello,
At this time, i don't "really need this bandwith".
But it will be when my new backup server will arrive with good disks.
iperf in the same interface or accross vlan doesn't change anything in term of bandwith ( i tested it ).
the Opnsense VM have 2 nics. ( which is phyiscally, the same 25Gb/s interface ).
one dedicated for the "upstream", the other for the internal with vlan.
When there is no trafic, my idle is never at 100%, but around 65-85%
6 processes: 1 running, 5 sleeping
CPU: 0.0% user, 0.0% nice, 0.0% system, 22.5% interrupt, 77.5% idle
Mem: 67M Active, 153M Inact, 113M Laundry, 1332M Wired, 56K Buf, 277M Free
ARC: 937M Total, 168M MFU, 602M MRU, 1926K Anon, 19M Header, 144M Other
662M Compressed, 1674M Uncompressed, 2.53:1 Ratio
Swap: 8192M Total, 8540K Used, 8184M Free
When i run an iperf from one ip of my vlan, to the ip of the opnsense ( so, no interval-routing )
6 processes: 1 running, 5 sleeping
CPU: 0.8% user, 0.0% nice, 0.0% system, 99.2% interrupt, 0.0% idle
Mem: 67M Active, 153M Inact, 113M Laundry, 1336M Wired, 56K Buf, 272M Free
ARC: 936M Total, 167M MFU, 604M MRU, 190K Anon, 19M Header, 144M Other
662M Compressed, 1675M Uncompressed, 2.53:1 Ratio
( at this point, the bandwith of the iperf is at 2.3Gb/s ).
Best regards,
I don't see 8GB of RAM in that output. 2GB is not enough. You end up consuming some swap!
The next step to identify the devices generating interrupts is:
'vmstat -i' (aggregated) and 'systat -vmstat' (live)
I'm still a bit fuzzy with your setup.
Single NIC OPN connected to a switch port (trunk). Another port of the switch is going towards an internet gateway, the others are LAN ports?
A backup server will be connected to one of these LAN ports.
What are you going to backup? The only traffic hitting OPN is LAN <-> WAN.
The rest (apart from super low bandwidth DHCP/DNS/...) is entirely handled by the switch.
Other VMs might use that NIC, but that's also not a concern for OPN.
Quote from: EricPerl on January 26, 2025, 09:06:58 PMI don't see 8GB of RAM in that output. 2GB is not enough.
Especially so if you are running ZFS which apparently you do. That's a good idea, generally, but 2 G is not nearly enough for a ZFS system to just boot and idly twiddle its thumbs. 4 G minimum in my experience.