Messages

OK, tried again after the upgrade to 23.1.4 (strongswan 5.9.10_1) did not fix the problem for me:

I had to manually restart the IPSec service (which the update apparently did not do), then it worked again.

A regression in the eap-tls plugin of Strongswan 5.9.10+ prevents EAP-TLS authentication. The log shows:

Code Select Expand

11[IKE] <con1|2> verification of AUTH payload with EAP MSK failed

Details: https://github.com/strongswan/strongswan/discussions/1613

Remedy: Downgrade to Strongswan 5.9.9_1 via

Code Select Expand

opnsense-revert -r 23.1.1 strongswan

But keep in mind that this version is affected by vulnerability CVE-2023-26463.

With this pull request merged, explanations are in place which should help to avoid being caught in this trap. Expect this to arrive with OPNsense 20.7.3.

EDIT: Maintainers have chosen to dismiss the above changes. As I cannot see how to contribute constructively without seeing rejects on seemingly arbitrary grounds, I can only hope that someone else will step in and improve usability.

I have just opened a feature request for the DNS blacklist case.

Another entry in OPNsense Unbound Custom options, which I have been using for diagnosis, is this one:

Code Select Expand

log-queries: yes

Could this be achieved currently without using Custom options?

[Interfaces > Settings]

It turned out that the most important setting change to avoid total loss of network connectivity was:

• In Interfaces > Settings set VLAN Hardware Filtering to Disable VLAN Hardware Filtering

This is the helptext in OPNsense for IPS Mode:
   
Enable protection mode (block traffic).
Before enabling, please disable all hardware offloading first in advanced network.

Thanks for pointing to this, but I'd like to respectfully disagree:

Interfaces > Settings contains three items which explicitly mention the term offload:

Code Select Expand


Hardware CRC Disable hardware checksum offload
Hardware TSO Disable hardware TCP segmentation offload
Hardware LRO Disable hardware large receive offload


The language used for the item

Code Select Expand


VLAN Hardware Filtering
does not fit into this category. While of course any kind of hardware processing could be labelled as offloading, I had understood the hint you were mentioning as referring to the upper three, more CPU-intensive items only. Another point is that the former three items are disabled by default while the latter is set as "Leave default" without documenting which default this refers to.

The above combination of factors unfortunately lead to a less-than-optimal out-of-box experience and somewhat time-consuming consequences. So I'd suggest the following changes for better usability.

Rename the settings as follows, converting each one to a positive language option with a checkbox, dropping the prior choice of "Leave default" for VLAN Hardware Filtering:

Code Select Expand


Enable Hardware CRC checksum offload             [ ]
Enable Hardware TSO (TCP segmentation offload)   [ ]
Enable Hardware LRO (large receive offload)      [ ]
Enable Hardware VLAN Filtering                   [ ]


Change the help text for IPS mode from

• "Before enabling, please disable all hardware offloading first in advanced network." to
• "Before enabling, please ensure that all Enable Hardware ... options in Interfaces > Settings are turned off. Otherwise you might experience a total loss of network connectivity on the router.

Any opinions on that?

[Interfaces > Settings]

After some experimentation I was able to successfully run Suricata in IPS mode with local VLANs on OPNsense 19.7.

Hardware tested:

• ZOTAC ZBOX PRO CI329 nano: Intel Celeron N4100, 2 x Realtek PCIe GBE (re), 8 GB RAM
• Thomas Krenn LES v3: Intel Celeron N3160, 2 x Intel i211AT (igb), 4 GB RAM

Topology:

• WAN: pppoe on igb1/re1
• LAN on igb0/re0 (not used directly)
• VLANS with LAN as parent (all internal hosts connect to one of the VLANs)

The primary problem I have experienced was the total loss of network connectivity (on all interfaces) when switching from IDS to IPS mode. Several workarounds posted elsewhere did not solve the problem but rather introduced new ones (cf. this posting).

It turned out that the most important setting change to avoid total loss of network connectivity was:

• In Interfaces > Settings set VLAN Hardware Filtering to Disable VLAN Hardware Filtering

Other configuration details:

Services > Intrusion Detection > Administration - Settings (in advanced mode):

• Enabled: (checked)
• IPS mode: (checked)
• Promiscuous mode: (checked)
• Pattern matcher: Hyperscan
• Interfaces: LAN
• Home networks: 192.168.0.0/16 10.0.0.0/8 172.16.0.0/12
• Log package payload: (checked)
• (Other settings left at their defaults.)

I have downloaded and enabled all rules offered by OPNsense with actions set to Drop, except for ET emerging-policy (downloaded and enabled, but action unchanged). Some individual rules were then disabled as deemed necessary.

In the above setting, Suricata did block VLAN traffic and reported "SID 7999999: OPNsense test eicar virus" on the "Alert" tab when running this test on an internal Linux host (on a VLAN):

Code Select Expand

curl http://malware.wicar.org/data/eicar.com > /dev/null

So far everything seems to run pretty stable with a scheduled reboot every 24 hours.

Diving a bit deeper, this is what is seems like:

• Current OPNsense kernels use drivers from the Realtek website. These are based on an old implementation of the FreeBSD re driver, which does not support netmap.
• When Suricata starts up in IPS mode and tries to use netmap, it fails with the Realtek drivers (while the standard FreeBSD re driver would provide the necessary netmap support).
• Even in non-IPS mode, Suricata would sometimes stop working until restarted. On some occasions, log messages "<Error> -- [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2" appeared in the log at very high frequency. In these cases, stopping Suricata took quite a while.
• My personal impression from observed stability problems and looking at the code is that the Realtek drivers have serious quality issues and should probably not be used in routers.

I thought I'd put this here for reference.

Tried Suricata in this configuration with stable/netmap kernels:

• OPNsense 19.1.9-amd64 on a ZBOX PRO CI329 nano (re0, re1: Realtek PCIe GBE)
• Suricata enabled, IPS mode enabled, Promiscuous mode disabled, Interfaces: LAN
• Hardware CRC, TSO, LRO disabled: all checked (disabling hardware offloading)
• sysctl dev.netmap.admode=1 (otherwise Suricata would block all traffic cf. https://redmine.openinfosecfoundation.org/issues/1688)
• There are VLANS configured on LAN (re0), but these have not been included in Suricata's interface list.
• Other options left at their defaults.

Suricata log with kernel FreeBSD 11.2-RELEASE-p10-HBSD  5e5adf26fc3(stable/19.1)  amd64:

Code Select Expand


Jul 5 18:55:19  suricata: [100282] <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#01-re0" failed to initialize: flags 0145
Jul 5 18:55:19 suricata: [101131] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [101130] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:53:39 suricata: [100282] <Warning> -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder.<proto>.<event>' to 'decoder.event.<proto>.<event>'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml.
Jul 5 18:53:39 suricata: [100102] <Notice> -- This is Suricata version 4.1.4 RELEASE


Suricata log with kernel FreeBSD 11.2-RELEASE-p9-HBSD  4ea457eb7b8(master)  amd64

Code Select Expand


Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#01-re0" failed to initialize: flags 0145
Jul 5 19:03:11 suricata: [100215] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100214] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:01:32 suricata: [100103] <Warning> -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder.<proto>.<event>' to 'decoder.event.<proto>.<event>'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml.
Jul 5 19:01:31 suricata: [100168] <Notice> -- This is Suricata version 4.1.4 RELEASE

So Suricata did not start up in either configuration.

Is Suricata expected to work in IPS mode in any of these configurations with the new kernel? Anything else I could try to improve the situation?