High CPU usage: Xeon E3-1265L V2

[fdiskc2000]

Hi, i´m actually using OPNsense 19.7.2 bare metal on an DELL R210i with 8GB RAM and this Xeon E3-1265L V2.

Thought could be overkill, but better to have to much than less performance.
Last month we got our new Fiber 600/300 and this OPNsense is working as router/firewall.

Of course to check what i get, i´m running speedtest.net from time to time. And everytime i do this i see high CPU Usage between 60-100%

Is this totally all 4 cores, or only one core maxing?

If this is really 100% of this CPU i´m wondering why this is so high.

Can someone give me a hint? I epexted this CPU to have more than enough power.

[opnfwb]

Suricata may be a major contributor to the CPU spikes depending on what it is configured to do. Also it looks like you're using OpenVPN? Make sure you're using AES-NI for crypto acceleration on those VPN tunnels, otherwise you can see increased CPU utilization if you're pushing 600mbits through the VPN tunnels.

Can you try a speed test with Suricata disabled and see if this impacts the CPU usage?

[fdiskc2000]

Hi thanks for your reply.

VPN is active, but no connection at the moment.
Disabled suricata, ntopng and monit but CPU usage is still 100% while speedtest is down/uploading.
no difference.

Is this really so CPU hungry? Can´t imagine.
Any idea where to look into?

The CPU is 7456 points at passmark, so it´s not so powerless?

[opnfwb]

Open an SSH session while running speed tests and run "top -aSCHIP". You can watch which process(s) is eating up CPU. Depending on what that shows will determine what the next steps are to see if it can be resolved.

[fdiskc2000]

Seems a reboot helped to reduce load. Don´t know.
Actually spikes come to 50-60%
What could be expected with this CPU?

[opnfwb]

That's a lot of interrupt usage. It looks like the 10GB Intel card is doing alright but the Broadcom card that is in that system is only using a single interrupt (probably one of the onboard NICs?). That seems to be a least one of the bottlenecks. As you can see, Suricata is also taking a significant CPU load, and so is NTOPng.

Can you post the output of: dmesg | grep -i msi

[fdiskc2000]

Yes you are right: 10GB Intel is a x520, rest is onboard from that DELL R210II (Broadcom)

dmesg:
ix0: Using MSI-X interrupts with 9 vectors
ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (7.10.0); Bufs (RX:2;TX:2;PG:; Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.13)
ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (7.10.0); Bufs (RX:2;TX:2;PG:; Flags (SPLT|MSI|MFW); MFW (NCSI 2.0.13)

What does it tell us? ;-)

[opnfwb]

If you're trying to max throughput I would expect the NIC to be using MSI-X and multiple queues to spread the packet load across multiple threads. This is what the 10GB Intel adapter is doing.

Generally on BSD, Broadcom drivers don't offer the tuning available to Intel drivers. So my first recommendation would be to try this with an Intel dual port or quad port NIC.

However, we can try a few things on the Broadcom right now and see if it helps.
First run, and copy the results of these commands:

sysctl hw.bce

vmstat -i

Then, run the following commands:

sysctl hw.bce.msi_enable=0

sysctl hw.bce.tso_enable=0

Try re-running your load tests with TSO and MSI disabled for BCE, some people report that this helps (even though it's technically less efficient). Also, you can try enabling them if they are already disabled (you can tell this from running the first command above, sysctl hw.bce, if they are already at a 0 value then try changing them to 1 and test.

[fdiskc2000]

Code: [Select]

sysctl hw.bce
hw.bce.rx_ticks: 18
hw.bce.rx_ticks_int: 18
hw.bce.rx_quick_cons_trip: 6
hw.bce.rx_quick_cons_trip_int: 6
hw.bce.tx_ticks: 80
hw.bce.tx_ticks_int: 80
hw.bce.tx_quick_cons_trip: 20
hw.bce.tx_quick_cons_trip_int: 20
hw.bce.strict_rx_mtu: 0
hw.bce.hdr_split: 1
hw.bce.tx_pages: 2
hw.bce.rx_pages: 2
hw.bce.msi_enable: 1
hw.bce.tso_enable: 1
hw.bce.verbose: 1


Code: [Select]

vmstat -i
interrupt                          total       rate
irq20: ehci0                     4982380          6
irq23: ehci1                     2351854          3
cpu0:timer                      85154548        104
cpu1:timer                      73024686         89
cpu4:timer                      77078037         94
cpu2:timer                      75039363         91
cpu5:timer                      74960795         91
cpu3:timer                      74592444         91
cpu7:timer                      75875867         93
cpu6:timer                      77778230         95
irq264: ix0:q0                   5731562          7
irq265: ix0:q1                   2844164          3
irq266: ix0:q2                   5737378          7
irq267: ix0:q3                   2941838          4
irq268: ix0:q4                   7090296          9
irq269: ix0:q5                   3092327          4
irq270: ix0:q6                   2624506          3
irq271: ix0:q7                   2074652          3
irq272: ix0:link                       4          0
irq273: bce0                    11023412         13
irq274: bce1                    11336488         14
irq275: ahci0                   13841993         17
Total                          689176824        840
Means MSI / TSO are enabled, right?

Try to disable this:

Code: [Select]

# sysctl hw.bce.msi_enable=0
sysctl: oid 'hw.bce.msi_enable' is a read only tunable
sysctl: Tunable values are set in /boot/loader.conf

# sysctl hw.bce.tso_enable=0
sysctl: oid 'hw.bce.tso_enable' is a read only tunable
sysctl: Tunable values are set in /boot/loader.conf

Is disableing this on loader.conf harmless or could the server get stuck with it?

[opnfwb]

Disabling shouldn't harm anything. Create this file: /boot/loader.conf.local

Input those two lines in the loader.conf.local file. Best practice is to NOT modify loader.conf because it will be overwritten on an upgrade. loader.conf.local will not be overwritten and your customizations will be retained during upgrades.

You can also do some Broadcom driver tuning by increasing pages and adding these lines to loader.conf.local:

sysctl hw.bce.tx_pages=8
sysctl hw.bce.rx_pages=8

You can try modifying the RX ticks and Quick Cons values too if you like. All of this is reversible (just delete the lines you don't want from loader.conf.local and reboot). The bad news is after all of this, we're running out of tuning to help with your CPU usage. So if these items don't make a significant impact we can't do much else short of trying a new NIC chipset.

[fdiskc2000]

Ok, thanks for your help anyway. I´ll keep you posted.

load.conf.local ist loaded automatically or does it need to be refered to from loader.conf

[opnfwb]

loader.conf.local is parsed automatically at boot. I've never had to make modifications to loader.conf.

Just create loader.conf.local and input any changes you want and reboot. You can test if the changes were applied at boot by running sysctl hw.bce and check if the values you input in loader.conf.local are now applied after a reboot.