[SOLVED] Suricata slows outbound traffic by a factor of 10 on KVM/virtio

[deajan]

Hello,

I'm running anOPNsense 23.1.6 virtualized via KVM on AlmaLinux 9.1, on a Xeon Gold 6312U.
All nics are virtio, all hardware accelerations are disabled in OPNsense.
So far so good, without using Suricata, I get about 850Mbits in/out speeds:

[root@virtual_guest_client ~]# ./speedtest -s 20411

   Speedtest by Ookla

      Server: Quantcom, a.s. - Prague (id: 20411)
         ISP: Ouiheberg Sarl
Idle Latency:    29.01 ms   (jitter: 0.12ms, low: 28.84ms, high: 29.11ms)
    Download:   849.25 Mbps (data used: 1.2 GB)
                 28.96 ms   (jitter: 1.31ms, low: 27.73ms, high: 38.89ms)
      Upload:   903.78 Mbps (data used: 1.3 GB)
                 29.80 ms   (jitter: 0.86ms, low: 28.38ms, high: 36.39ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/29c7f928-abc2-41c8-ba13-59359d3b18c7

If I happen to enable suricata, using ET Telemetry ruleset, IPS mode with Hyperscan engine, upload speed are terrible:

[root@virtual_guest_client ~]# ./speedtest -s 20411

   Speedtest by Ookla

      Server: Quantcom, a.s. - Prague (id: 20411)
         ISP: Ouiheberg Sarl
Idle Latency:    29.72 ms   (jitter: 0.17ms, low: 29.45ms, high: 29.85ms)
    Download:   875.42 Mbps (data used: 1.6 GB)
                 28.11 ms   (jitter: 3.05ms, low: 27.52ms, high: 291.24ms)
      Upload:    67.82 Mbps (data used: 55.4 MB)
                 29.38 ms   (jitter: 0.48ms, low: 28.12ms, high: 30.38ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/9f80a999-38de-4e4e-bf5d-31d3352d0726

I've played with various things in order to find what the culprit could be:

- Using E1000 nics was disastrous, got 700% cpu usage on host when OPNsense was idle, instead of roughly 5%, also got some packet loss
- Using a Q35 machine instead of a 440fx with qemu: Did not change alot
- Playing with tuned profiles: Got 30% better upload speed when using performance cpu governor instead of conservative
- Isolated CPUs from host and made the OPNsense cpu affinity use those cpus: Did not change alot

So it looks indeed that my problem is CPU bound, since using a better cpu governor helps getting better speeds.
Nevertheless, my host has a 2.4Ghz Xeon CPU, with 3.6Ghz turbo frequency.
The CPU cores were indeed clocked at 3.6Ghz while performing the tests.

I cannot imagine Suricata being so hungry that 8 x 3.6Ghz CPUs won't suffice for having 1GB throughput.

Can someone confirm if this has something todo with KVM ?
If so, any advices perhaps ?

Best regards.

[deajan]

Other things I tried without success:

- Setting echo 10000 > /sys/module/kvm/parameters/halt_poll_ns on KVM host, like described in https://forum.suricata.io/t/cpu-usage-of-version-6-0-0/706/22
- Tuning KVM to near realtime, like described in https://libvirt.org/kbase/kvm-realtime.html
- Reading https://forum.opnsense.org/index.php?topic=6190.15 - no I don't have a proxy on my OPNSense. It's a vanilla setup with only Qemu agent and suricata configured
- Reducing ruleset, got down from 140k rules to just ET telemetry, 22k rules, no noticeable difference
- Read the pinned topic at https://forum.opnsense.org/index.php?topic=6590.0 - Since I don't have physical NICs, most hacks don't apply to me
- Tried to apply various parameters from https://forum.opnsense.org/index.php?topic=6590.msg110020#msg110020 - No improvements

One thing, if I happen to disable IPS, but keeping IDS enabled, I get good speeds again, so the problem can't be blamed on the capture driver I guess.

[deajan]

I've seen that latest OPNense version upates suricata from 6.0.9_1 to 6.0.11_1.
Applied the upgrade, but still got the same performance issue.

What puzzles me is that another AlmaLinux 8.x server with a KVM virtualized OPNSense keeps good outgoing performance, regardless of the IPS setting.

Any guidance would be appreciated

[featheredfifth]

I've seen that latest OPNense version upates suricata from 6.0.9_1 to 6.0.11_1.
Applied the upgrade, but still got the same performance issue.

geometry dash world

What puzzles me is that another AlmaLinux 8.x server with a KVM virtualized OPNSense keeps good outgoing performance, regardless of the IPS setting.

Any guidance would be appreciated

Have any solution for this issue?

[deajan]

Never really found the solution.
All I can say is that the other OPNSense KVM instances I have use PCI passthrough and don't suffer the suricata performance impact, so the problem is definitly KVM virtio related.

I've rechecked with current OPNsense 23.7.11-amd64 and qemu-kvm-8.0.0-16.el9_3.1.alma.1.x86_64 and the results are still the same.
Enable suricata and upload speeds are divided by an order of magnitude.

[deajan]

I've also tried setting the virtio net driver queue number, eg:
```
<interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
      <driver name='vhost' queues='N'/>
</interface>
```
This gives me roughly a x2 boost, so we're definitly dealing with a virtio performance issue.

I also read somewhere that without hardware offloading, we're stuck with host CPU performance for network, which should be quite bad.
This computes since if I change the cpu power profile, I squeeze another percentage of speed improvement out as I said earlier.

[deajan]

After more digging, it looks like this is the actual culprit:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059

Unless some dev guru at FreeBSD can fix hardware offloading with virtio... I'd never advise virtio drivers.
PCI Passthrough is the way (when possible).

[JL]

Having run Suricata on OPNSense with virtio since many years I do not have such issue.
The internet line is 100Mbps but the lan is set to 1Gbps.

Here's my best guess.

Don't try and "tweak" network drivers, this is overwritten in many cases due to limitations in the driver(s) and such.
Yes, especially not with e1000 and related cards.

I've had long time issues with Suricata uptime until I finally had some time and fixed the MTU by setting dev.netmap.bufsize to the relevant MTU value.

Also consider evaluating the MTU for the bridge interfaces. To my understanding it is best to have large MTU for Gbps networks.

With some Linux you (at least this used to be so) may need to tweak sysctl settings to allow for large transfers.
I assume you've checked that ?

Also, are there any console message visible ?

Wrong MTU size for example will throw an error like: netmap_buf_size_validate error: large MTU (8192) needed but igb1 does not support NS_MOREFRAG

[deajan]

So I finally tried a last solution posted on a FreeBSD forum https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059

Setting the following in `/boot/loader.conf` solved the issue:

Adding parameters to /boot/loader.conf

Code: [Select]

#hw.vtnet.X.tso_disable="1"
hw.vtnet.tso_disable="1"
hw.vtnet.lro_disable="1"
#hw.vtnet.X.lro_disable="1"
hw.vtnet.csum_disable="1"
#hw.vtnet.X.csum_disable="1"

I've jumped from remote uploads at 20-30Mbits to a whopping 300Mbits ^^
I've previously tried this with ethtool without success.
Also, I had to reboot in order for the changes to take effect.