[SOLVED] [Suricata] Suricata dropping traffic with IPS.

[xupetas]

 Hello All.

I've installed the latest version of opnsense and tryed to run suricata.
My HW is a 3 vCore QEMU/KVM (tryed on qemu 2.7 and 2.9) with 2GB of ram and several VIRTIO NICs.

I've pulled the latest rules, and checked if there was updates to be installed.
Both suricata and opnsense are on the latest version.

When i boot suricata in IPS configuration, on the WAN interface, i loose conectivity.
Here's a ping form the firewall to the next hop to demonstrate what appends:

64 bytes from 192.168.1.254: icmp_seq=0 ttl=64 time=1.135 ms
64 bytes from 192.168.1.254: icmp_seq=1 ttl=64 time=0.991 ms
64 bytes from 192.168.1.254: icmp_seq=2 ttl=64 time=0.595 ms
64 bytes from 192.168.1.254: icmp_seq=3 ttl=64 time=0.749 ms
64 bytes from 192.168.1.254: icmp_seq=4 ttl=64 time=0.828 ms
64 bytes from 192.168.1.254: icmp_seq=5 ttl=64 time=0.803 ms
64 bytes from 192.168.1.254: icmp_seq=6 ttl=64 time=0.766 ms
64 bytes from 192.168.1.254: icmp_seq=7 ttl=64 time=68606.310 ms < Suricata boot
64 bytes from 192.168.1.254: icmp_seq=8 ttl=64 time=68300.215 ms
64 bytes from 192.168.1.254: icmp_seq=10 ttl=64 time=67848.197 ms
64 bytes from 192.168.1.254: icmp_seq=14 ttl=64 time=68108.471 ms
64 bytes from 192.168.1.254: icmp_seq=17 ttl=64 time=67052.875 ms
64 bytes from 192.168.1.254: icmp_seq=18 ttl=64 time=67528.361 ms
64 bytes from 192.168.1.254: icmp_seq=19 ttl=64 time=68002.670 ms
64 bytes from 192.168.1.254: icmp_seq=22 ttl=64 time=67999.273 ms
64 bytes from 192.168.1.254: icmp_seq=23 ttl=64 time=67995.854 ms

There are NO rules loaded for this test, I just started the daemon.

If i stop the IPS mode (active defense) it goes back to normal:

64 bytes from 192.168.1.254: icmp_seq=10 ttl=64 time=38098.266 ms
64 bytes from 192.168.1.254: icmp_seq=15 ttl=64 time=36568.727 ms
64 bytes from 192.168.1.254: icmp_seq=16 ttl=64 time=36987.789 ms
64 bytes from 192.168.1.254: icmp_seq=20 ttl=64 time=37390.267 ms
64 bytes from 192.168.1.254: icmp_seq=21 ttl=64 time=36703.024 ms
64 bytes from 192.168.1.254: icmp_seq=24 ttl=64 time=36385.351 ms
64 bytes from 192.168.1.254: icmp_seq=26 ttl=64 time=51639.619 ms
64 bytes from 192.168.1.254: icmp_seq=27 ttl=64 time=52612.838 ms
64 bytes from 192.168.1.254: icmp_seq=29 ttl=64 time=51815.528 ms
64 bytes from 192.168.1.254: icmp_seq=30 ttl=64 time=51632.411 ms
64 bytes from 192.168.1.254: icmp_seq=32 ttl=64 time=50433.146 ms
64 bytes from 192.168.1.254: icmp_seq=85 ttl=64 time=0.504 ms
64 bytes from 192.168.1.254: icmp_seq=86 ttl=64 time=0.473 ms
64 bytes from 192.168.1.254: icmp_seq=87 ttl=64 time=0.462 ms
64 bytes from 192.168.1.254: icmp_seq=88 ttl=64 time=0.404 ms
64 bytes from 192.168.1.254: icmp_seq=89 ttl=64 time=0.502 ms
64 bytes from 192.168.1.254: icmp_seq=90 ttl=64 time=0.444 ms
64 bytes from 192.168.1.254: icmp_seq=91 ttl=64 time=0.474 ms

Am i missing something or is this a bug?
I need the IPS active to do active defense on my system?
Or can it be run with active defence on legacy configuration thru pcap thus being only and IDS and not a IPS?

Thanks,
Nuno

[scrensen]

Hi,

I have the exact same 'issue'.

And do you also see a drop in speed? I have a 400 Mb/s subscription, but with Suricata enabled (in IPS mode with 9 rulesets) I get around 120 Mb/s. And when I disable the Intrusion Detection service it immediately goes back to around 400 Mb/s

[franco]

Hi Nuno,

See if e1000 driver emulation helps... IPS mode is built around Netmap, which like its cousin DPDK is strongest on actual hardware. VMs can be tricky or less performant.

@scrensen, that does not seem to be the same issue. I'm sure you have a number of rules enabled and packets are not dropped. Not knowing anything about your hardware and software setup, I'd say it looks to be within the realm of general possibility.


Cheers,
Franco

[xupetas]

I have changed form the virtio vnic to the e1000 vnic.
It works perfectly now as expected.

Thanks for your help and feedback!

[Fabio83]

Hello.

I have exactly the same issue with my virtual OPNsense, even if I switch to E1000. All of the activated Interfaces (inside IDS configuration) won't work if I enable IPS.

This behaviour I have with VIRTIO & E1000. Just with VMXNET3 it seems to works. But here I loose ~70% of the physical Bandwidth after activating IDS/IPS. (even if I have increased CPU Cores & Memory from this Virtual Machine)

It would be great if you have any Idea how to realize OPNsense inside a virtual Machine with a good performance and activated IPS.


Thanks,
Fabio

[missionleben]

Same thing with us, I'm afraid. No problem without ips. With ips massive problems. No matter which driver. (tested virtio and e1000). I'm at a loss, too. It works in front of all with pfsense. Been wanting to replace our big pfsense for a long time. Doesn't make any real sense without ips.

Regards

Christian

Gesendet von meinem Redmi Note 4 mit Tapatalk

[franco]

Er, does this pfSense do native IPS (netmap) mode on the same virtualisation host? If yes, which version?

[xupetas]

Hello all,

I am running my opnsense firewall cluster on two differente versions of KVM/QEMU:

qemu-2.7.0-360
qemu-2.9.0-385

Both of them work pefectly now with IPS and the em0 on the interface that is going to be using the IPS feature (on my case the WAN)

On wich virtualization engine you are having issues?

Thanks!
Nuno

[Julien]

For me the same
its kills the connections from 1Gbps/s to 200Mbps.
I am on a hardware
hardware offload is disabled but its still killing my connection.

[Fabio83]

Hello Nuno.

Thank you for this information.

I am currently running pve-qemu-kvm: 2.9.0-pve4 @ my PROXMOX PVE Host.


Fabio

[xupetas]

For me the same
its kills the connections from 1Gbps/s to 200Mbps.
I am on a hardware
hardware offload is disabled but its still killing my connection.

Hello Julien,

Is the difficulty being shown at any speed? Or is only felt when you reach 200mbps?

Thanks!

[Fabio83]

Hello Julien,

Is the difficulty being shown at any speed? Or is only felt when you reach 200mbps?

Thanks!

In my environment I tested it via iperf to a System in another Subnet or over WAN:
Virtio and IPS disabled: ~900Mbit/s
E1000 and IPS disabled: ~200-250Mbit/s
VMXNET3 and IPS disabled: ~500Mbit/s
VMXNET3 and IPS enabled: ~300-400Mbit/s

Grretings,
Fabio

[xupetas]

Hello Julien,

Is the difficulty being shown at any speed? Or is only felt when you reach 200mbps?

Thanks!

In my environment I tested it via iperf to a System in another Subnet or over WAN:
Virtio and IPS disabled: ~900Mbit/s
E1000 and IPS disabled: ~200-250Mbit/s
VMXNET3 and IPS disabled: ~500Mbit/s
VMXNET3 and IPS enabled: ~300-400Mbit/s

Grretings,
Fabio

Hello again.

My problem was not the same as yours. My connections simply dropped dead. No even ping would pass.
It appears that is issue on your case is the vCPU of the opnsense appliance is not strong enough to handle all the traffic and IPS at the same time.
Can you add other vCPU's to your vm, and pin them with exclusivity to a physical CPU on the host and re-check?

Thanks

[Fabio83]

Hello Julien,

Is the difficulty being shown at any speed? Or is only felt when you reach 200mbps?

Thanks!

In my environment I tested it via iperf to a System in another Subnet or over WAN:
Virtio and IPS disabled: ~900Mbit/s
E1000 and IPS disabled: ~200-250Mbit/s
VMXNET3 and IPS disabled: ~500Mbit/s
VMXNET3 and IPS enabled: ~300-400Mbit/s

Grretings,
Fabio

Hello again.

My problem was not the same as yours. My connections simply dropped dead. No even ping would pass.
It appears that is issue on your case is the vCPU of the opnsense appliance is not strong enough to handle all the traffic and IPS at the same time.
Can you add other vCPU's to your vm, and pin them with exclusivity to a physical CPU on the host and re-check?

Thanks

Yes, I have the same issue with E1000 and VirtIO, if IPS is enable -> nothing works!
And of course - I still played with the amount of vCores and with the emulated CPU Type. But it is always the same with E1000 and VirtIO.
I am running my Installation currently with VMXNET3, because I don't want to give up IPS...

[xupetas]

Hello Julien,

Is the difficulty being shown at any speed? Or is only felt when you reach 200mbps?

Thanks!

In my environment I tested it via iperf to a System in another Subnet or over WAN:
Virtio and IPS disabled: ~900Mbit/s
E1000 and IPS disabled: ~200-250Mbit/s
VMXNET3 and IPS disabled: ~500Mbit/s
VMXNET3 and IPS enabled: ~300-400Mbit/s

Grretings,
Fabio

Hello again.

My problem was not the same as yours. My connections simply dropped dead. No even ping would pass.
It appears that is issue on your case is the vCPU of the opnsense appliance is not strong enough to handle all the traffic and IPS at the same time.
Can you add other vCPU's to your vm, and pin them with exclusivity to a physical CPU on the host and re-check?

Thanks

Yes, I have the same issue with E1000 and VirtIO, if IPS is enable -> nothing works!
And of course - I still played with the amount of vCores and with the emulated CPU Type. But it is always the same with E1000 and VirtIO.
I am running my Installation currently with VMXNET3, because I don't want to give up IPS...

It is normal a drop in traffic when using IPS. It takes the CPU time to process and validade the data.
It's the same on every IDS/IPS system.

On this case, the usage that suricata does of the netmap driver capability makes it more noticeable.