Messages

at this moment i disabled Suricata

To my surprise I just ended up here as well.
After experiencing a lot of strange and unstable behavior I finally pulled the plug.

It may work better as a dedicated Linux based VM but on OPNsense/BSD for the time being the cost in terms of stability and performance is currently too high for me with my virtual infrastructure setup.

Also the whole network latency seems to have improved perceiveably w/o suricata.

A significant part of the suricata rules seem to be DNS and IP blacklists anyway which can easily be implemented by other means.

[destination]

Hi,

In the proposed configuration it's currently destination only.
Having it in source and destination would require to set up two separate rules.

For most simple configurations destination should be good enough to catch potential outgoing traffic if you block incoming traffic from WAN already - which probably most simple setups do.

If you have a more advanced set up where you allow incoming traffic from WAN, i.e. for a VPN or other services you may want to have an additional rule to block incoming traffic by source.

That would be my current understanding of how it works.
Does it make sense or have I overlooked something?

I'm coming from there, respectively had both activated which drags down the performance even more to around 3 Gbps here.

I somehow have more confidence in the

Code Select Expand

ET Pro Telemetry ruleset and this is why I decided to go with suricata only for the time being.

I may change my mind again at some point because the usability and out of the box reporting capabilities of suricata leave a lot to desire when compared to sensei.

Concerning the ruleset I think ET Pro Telemetry is superior.

I am able to push through ~7 Gbps peak with Suricata enabled, virtualized with an intel X550-T2 with SR-IOV enabled.

The whole setup is still sort of experimental as I have some rather strange SR-IOV issues with OPNsense but with a bit of tinkering it works.

Important performance considerations were:
- In the KVM config set CPU type to host, to maximally benefit from CPU features/optimizations (I have an 11th Gen Intel(R) Core(TM) i5-11600 @ 2.80GHz)
- Use SR-IOV (I got worse performance passing through the whole PCI device - I still have no idea why ~3 Gpbs)
- Pattern Matcher: Hyperscan

No, it wasn't working with 21.1 as expected either. I wanted to see if there's an improvement with 21.7 but there's not.

As I do not see much helpful log output on 21.1/7 I was wondering if the additional info that is outputted on 21.7

Code Select Expand

c055.723348 [ 853] iflib_netmap_config       txr 2 rxr 2 txd 2048 rxd 2048 rbufsz 2048

Could give any hints regarding the potential root cause of the issue but it seems this is just additional information being outputted with 21.7.

So 21.1 and 21.7 behave the same with the exception of this additional info line in dmesg in 21.7.

Edit updated now to:

OPNsense 21.7.r2-amd64
FreeBSD 12.1-RELEASE-p19-HBSD
OpenSSL 1.1.1k 25 Mar 2021

This is Suricata version 5.0.7 RELEASE running in SYSTEM mode

Behavior is the same.

before:

OPNsense 21.7.r1-amd64
FreeBSD 12.1-RELEASE-p18-HBSD
OpenSSL 1.1.1k 25 Mar 2021

This is Suricata version 5.0.6 RELEASE running in SYSTEM mode

Thanks a lot Stephan for your helpful explanation.

The issue I'm currently facing is that with SR-IOV the VF interface I expose to the OPNsense VM gets kind of burned during boot/initialization and shortly thereafter stops working. The entries in dmesg are what I see when this is happening.

I worked around this by exposing 2 VF interfaces and after OPNsense has booted up I switch from one to the other. That way it then works but I don't understand why this behavior is occuring.

I have not yet found out what is causing the problems with the VF interface that is active during boot/initialization and was wondering whether these messages could give a clue as I do not see anything helpful in the logs otherwise. Seems that this is also not the case.
It is an OPNsense and maybe BSD specific issue that I was not able to reproduce with a Debian VM.

Thanks for your help anyway.

I'm seeing these in dmesg on 21.7.:

Code Select Expand


055.723348 [ 853] iflib_netmap_config       txr 2 rxr 2 txd 2048 rxd 2048 rbufsz 2048


(did not appear on 21.1 only UP/DOWN)
which might be related to my SR-IOV NIC issues.
Would anyone know what it is and by what it may be caused?

Code Select Expand


ixv1: link state changed to DOWN
ixv1: link state changed to UP
ixv1: link state changed to DOWN
ixv1: link state changed to UP
ixv1: link state changed to DOWN
ixv1: link state changed to UP
055.451495 [ 853] iflib_netmap_config       txr 2 rxr 2 txd 2048 rxd 2048 rbufsz 2048
055.452924 [ 853] iflib_netmap_config       txr 2 rxr 2 txd 2048 rxd 2048 rbufsz 2048
ixv0: link state changed to DOWN
ixv0: link state changed to UP
055.723348 [ 853] iflib_netmap_config       txr 2 rxr 2 txd 2048 rxd 2048 rbufsz 2048
ixv0: link state changed to DOWN
ixv0: link state changed to UP


Edit:
What is the intel driver version used in 21.7?
3.3.24 seems to be the most recent.

Cross posting this here for better discoverability:

Blocking malicious IPs with OPNsense using spamhaus droplists and dshield_30_days is actually quite easy.

How it's done:
➡️ https://www.allthingstech.ch/blocking-malicious-ips-with-opnsense/

Ok got it, thanks.

Now that I discovered the VF promiscuous mode limitation the only option that will work is indeed as you initially suggested to hand through the whole device/port.

Seems it's possible according to here:
https://downloadmirror.intel.com/14688/eng/README_3.3.24.txt

But I also stumbled across this:

Some notable limitations of the VF environment:
  * For security reasons, the driver is never permitted to be promiscuous, therefore a tcpdump will not behave the same with the interface.

Which seems to confirm my assumption related to promiscuous mode.
--> That means it will likely only work when passing through the whole device.

Thanks for your help nevertheless.

That will happen automatically with the switch from HardenedBSD to stock FreeBSD in 2022.

That I understood. But isn't it possible to pull are more recent driver and build a module as with Linux?
Or is with FreeBSD everything hardwired into the kernel?

[Assumption 1]

I think I identified a pattern and a rather insane workaround for the time being:

It seems that the VF interface that is present at OPNsense boot time quickly works (a few seconds) but then immediately after this gets somehow burned and becomes defunct.

My approach to mitigate this is atm to expose two VF PCI cards to OPNsense and after boot switch to the other one - insane but so far it works until next reboot when this is required again ;)

in dmesg it looks like this:

Code Select Expand


ixv0: permanently promiscuous mode enabled
ixv0: link state changed to DOWN
ixv0: link state changed to UP
ixv0: link state changed to DOWN
ixv0: link state changed to UP
ixv0: link state changed to DOWN
ixv0: link state changed to UP
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled

--> Switching interface to ixv1 here

ixv1: link state changed to DOWN
ixv1: link state changed to UP
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled


Assumption 1
Could it be related to promiscuous mode?
I see it happening for ixv0 but never for ixv1 "ixv0: permanently promiscuous mode enabled"
I assume this is triggered through the respective suricata setting and may not be executed properly again after changing the WAN interface assignment and therefore leave it working?

Assumption 2
Could this maybe be related to an old version of the intel driver being used with OPNsense?
Is it possible to pull in a newer version?

Thanks for your response. I may give 13/12.2 a try and observe.

SR-IOV would be just too nice, to give it up too early. And I'm afraid of running into the same issues passing along the whole card.

I'm mostly irritated by not finding any errors in dmesg and system.log

Hi,

I'm having inconsistent to erratic behavior with OPNsense as a VM guest in combination with SR-IOV.
I have SR-IOV enabled on the Proxmox host and also for a Debian guest - the latter as a verification.
The throughput increase on OPNsense with suricata enabled is an impressive factor 3-4 which makes SR-IOV worthwhile.

But the OPNsense guest sometimes doesn't want to run at all with the VF interface, sometimes runs fine for hours  and then the VF interfaces suddenly stops working for OPNsense - sometimes after a reboot or just out of the blue, for no obvious reason.

I have no clue what the cause of this inconsistent behavior is and I do not see anything meaningful in dmesg output or in /var/log/system.log or on the host besides sudden link state changes of the VF interface in the guest VM only.

On the Debian guest though everything keeps running smoothly all the time so the problem seems only to be related to the OPNsense guest.

Is this a known upstream FreeBSD issue or should this work in general with OPNsense?
Where could I look for helpful log data - dmesg and system.log have not proofed to be very helpful so far.

Thanks

Edit: Wrong forum this is on 21.1