Understand CPU USage

[henri9813]

Hello,

My hypervisor has 2 nics:
- 1x 1Gb/s
- 1x 25Gb/s connected to a switch ( which lead to my upstream ).


I use OPNsense 24.7.12_2-amd64 in a XCP-NG virtual machine which has 4c / 8Gb ram / 50Gb nvme.

The CPU of the server is AMD EPYC 4464P (3,7 GHz )

i'm in Hardware virtualization with paravirtualization drivers enabled (PVHVM) with Realtek 8139
i attached 2x the 25gb/s link ( 1 for wan, 1 for the local VLAN ).

I can't get better than 3Gb/s

Code Select Expand

[18:39 server-1 ~]# iperf3 -c 10.255.0.254
Connecting to host 10.255.0.254, port 5201
[  4] local 10.255.1.3 port 38188 connected to 10.255.0.254 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   351 MBytes  2.95 Gbits/sec  362    536 KBytes       
[  4]   1.00-2.00   sec   331 MBytes  2.78 Gbits/sec    4    582 KBytes       
[  4]   2.00-3.00   sec   354 MBytes  2.97 Gbits/sec   21    465 KBytes       
[  4]   3.00-4.00   sec   341 MBytes  2.86 Gbits/sec    5    556 KBytes       
[  4]   4.00-5.00   sec   310 MBytes  2.60 Gbits/sec   15    441 KBytes       
[  4]   5.00-6.00   sec   354 MBytes  2.97 Gbits/sec   64    644 KBytes       
[  4]   6.00-7.00   sec   345 MBytes  2.89 Gbits/sec    6    530 KBytes       
[  4]   7.00-8.00   sec   312 MBytes  2.62 Gbits/sec    7    671 KBytes       
[  4]   8.00-9.00   sec   348 MBytes  2.92 Gbits/sec   14    581 KBytes       
[  4]   9.00-10.00  sec   326 MBytes  2.74 Gbits/sec   50    571 KBytes       

When i run the iperf, the cpu is full !


When i disable the firewall in settings, the iperf is 2x more performant.

Code Select Expand

[18:45 server-1 ~]# iperf3 -c 10.255.0.254 -t 10000
Connecting to host 10.255.0.254, port 5201
[  4] local 10.255.1.3 port 38318 connected to 10.255.0.254 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   617 MBytes  5.17 Gbits/sec  464    742 KBytes       
[  4]   1.00-2.00   sec   540 MBytes  4.53 Gbits/sec   57    671 KBytes       
[  4]   2.00-3.00   sec   639 MBytes  5.36 Gbits/sec  118    522 KBytes       
[  4]   3.00-4.00   sec   646 MBytes  5.42 Gbits/sec   58    636 KBytes       
[  4]   4.00-5.00   sec   665 MBytes  5.58 Gbits/sec   35    599 KBytes       
[  4]   5.00-6.00   sec   699 MBytes  5.86 Gbits/sec  150    698 KBytes       
[  4]   6.00-7.00   sec   616 MBytes  5.16 Gbits/sec  128    702 KBytes       
[  4]   7.00-8.00   sec   692 MBytes  5.82 Gbits/sec  156    735 KBytes       
^C[  4]   8.00-8.53   sec   380 MBytes  6.03 Gbits/sec   50    509 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-8.53   sec  5.37 GBytes  5.40 Gbits/sec  1216             sender
[  4]   0.00-8.53   sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated


I precise that i also tried to remove ALL MY RULES, i don't see any difference in term of cpu usage.

Why do i use all theses CPU ?

And the final question, yesterday, after a ton of test ( which i don't note somewhere of course ) i figured to have my 17GB/s !!! , but after a reboot, my bandwith come back to 3GB/s...

Code Select Expand

[19:52 server-1 ~]# iperf3 -c 10.255.1.254
Connecting to host 10.255.1.254, port 5201
[  4] local 10.255.1.3 port 59300 connected to 10.255.1.254 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.80 GBytes  15.4 Gbits/sec  265    682 KBytes       
[  4]   1.00-2.00   sec  1.79 GBytes  15.3 Gbits/sec  245    743 KBytes       
[  4]   2.00-3.00   sec  1.88 GBytes  16.2 Gbits/sec  216   1014 KBytes       
[  4]   3.00-4.00   sec  1.91 GBytes  16.4 Gbits/sec  138   1.60 MBytes       
[  4]   4.00-5.00   sec  1.85 GBytes  15.9 Gbits/sec  153   1.98 MBytes       
[  4]   5.00-6.00   sec  2.00 GBytes  17.2 Gbits/sec  262    638 KBytes       
[  4]   6.00-7.00   sec  1.92 GBytes  16.5 Gbits/sec  351    944 KBytes       
[  4]   7.00-8.00   sec  1.78 GBytes  15.3 Gbits/sec  241   2.06 MBytes       
[  4]   8.00-9.00   sec  1.97 GBytes  16.9 Gbits/sec  240    655 KBytes       
[  4]   9.00-10.00  sec  1.58 GBytes  13.6 Gbits/sec  210   2.01 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  18.5 GBytes  15.9 Gbits/sec  2321             sender
[  4]   0.00-10.00  sec  18.5 GBytes  15.9 Gbits/sec                  receiver

When i run no trafic, my CPU usage is at 13%-20%.

Thanks !

[EricPerl]

I run OPNsense (4 cores, 8GB RAM, 64GB disk) virtualized on Proxmox on a N305 fanless NUC.

Looking at top at the console, the system is largely idle (sometimes completely) under background use:

Code Select Expand

last pid: 30707;  load averages:  0.22,  0.23,  0.21                                            up 9+23:37:29  15:00:58
90 processes:  1 running, 89 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

Under 300Mbps load (my ISP max), interrupts go up to ~5%.
Proxmox is pretty lightweight and has reasonable overhead (the above seems to translate to a mix of user/system ~8%).
I have no interest in pushing higher load (would force me to run some inter VLAN arbitrary test).

I'd check where the CPU is going. top is a start.
Do you actually need that kind of bandwidth (LAN/WAN or Inter-VLAN) or is it merely a test?

[EricPerl]

Oh, I forgot to mention it, your last test shows this:
[  4] local 10.255.1.3 port 59300 connected to 10.255.1.254 port 5201
where other tests had 10.255.0.254 for destination.

If that last test was within VLAN, and the others were not (i.e. inter-VLAN), that could explain the difference.
The VLAN traffic may not touch the router at all.

OPN is running off a single NIC with one VLAN for LAN and another for WAN?
I've never used that config. It's on my to-do...

[henri9813]

Hello,

At this time, i don't "really need this bandwith".

But it will be when my new backup server will arrive with good disks.

iperf in the same interface or accross vlan doesn't change anything in term of bandwith ( i tested it ).

the Opnsense VM have 2 nics. ( which is phyiscally, the same 25Gb/s interface ).

one dedicated for the "upstream", the other for the internal with vlan.

When there is no trafic, my idle is never at 100%, but around 65-85%

Code Select Expand

6 processes:   1 running, 5 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system, 22.5% interrupt, 77.5% idle
Mem: 67M Active, 153M Inact, 113M Laundry, 1332M Wired, 56K Buf, 277M Free
ARC: 937M Total, 168M MFU, 602M MRU, 1926K Anon, 19M Header, 144M Other
     662M Compressed, 1674M Uncompressed, 2.53:1 Ratio
Swap: 8192M Total, 8540K Used, 8184M Free


When i run an iperf from one ip of my vlan, to the ip of the opnsense ( so, no interval-routing )

Code Select Expand

6 processes:   1 running, 5 sleeping
CPU:  0.8% user,  0.0% nice,  0.0% system, 99.2% interrupt,  0.0% idle
Mem: 67M Active, 153M Inact, 113M Laundry, 1336M Wired, 56K Buf, 272M Free
ARC: 936M Total, 167M MFU, 604M MRU, 190K Anon, 19M Header, 144M Other
     662M Compressed, 1675M Uncompressed, 2.53:1 Ratio

( at this point, the bandwith of the iperf is at 2.3Gb/s ).

Best regards,

[EricPerl]

I don't see 8GB of RAM in that output. 2GB is not enough. You end up consuming some swap!

The next step to identify the devices generating interrupts is:
'vmstat -i' (aggregated) and 'systat -vmstat' (live)

I'm still a bit fuzzy with your setup.
Single NIC OPN connected to a switch port (trunk). Another port of the switch is going towards an internet gateway, the others are LAN ports?

A backup server will be connected to one of these LAN ports.
What are you going to backup? The only traffic hitting OPN is LAN <-> WAN.
The rest (apart from super low bandwidth DHCP/DNS/...) is entirely handled by the switch.
Other VMs might use that NIC, but that's also not a concern for OPN.

[Patrick M. Hausen]

I don't see 8GB of RAM in that output. 2GB is not enough.

Especially so if you are running ZFS which apparently you do. That's a good idea, generally, but 2 G is not nearly enough for a ZFS system to just boot and idly twiddle its thumbs. 4 G minimum in my experience.