Call for testing: New netmap enabled kernel

[Oliver]

Tried Suricata in this configuration with stable/netmap kernels:

• OPNsense 19.1.9-amd64 on a ZBOX PRO CI329 nano (re0, re1: Realtek PCIe GBE)
• Suricata enabled, IPS mode enabled, Promiscuous mode disabled, Interfaces: LAN
• Hardware CRC, TSO, LRO disabled: all checked (disabling hardware offloading)
• sysctl dev.netmap.admode=1 (otherwise Suricata would block all traffic cf. https://redmine.openinfosecfoundation.org/issues/1688)
• There are VLANS configured on LAN (re0), but these have not been included in Suricata's interface list.
• Other options left at their defaults.

Suricata log with kernel FreeBSD 11.2-RELEASE-p10-HBSD  5e5adf26fc3(stable/19.1)  amd64:

Code: [Select]

Jul 5 18:55:19  suricata: [100282] <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#01-re0" failed to initialize: flags 0145
Jul 5 18:55:19 suricata: [101131] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [101130] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:53:39 suricata: [100282] <Warning> -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder.<proto>.<event>' to 'decoder.event.<proto>.<event>'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml.
Jul 5 18:53:39 suricata: [100102] <Notice> -- This is Suricata version 4.1.4 RELEASE

Suricata log with kernel FreeBSD 11.2-RELEASE-p9-HBSD  4ea457eb7b8(master)  amd64

Code: [Select]

Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#01-re0" failed to initialize: flags 0145
Jul 5 19:03:11 suricata: [100215] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100214] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:01:32 suricata: [100103] <Warning> -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder.<proto>.<event>' to 'decoder.event.<proto>.<event>'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml.
Jul 5 19:01:31 suricata: [100168] <Notice> -- This is Suricata version 4.1.4 RELEASE
So Suricata did not start up in either configuration.

Is Suricata expected to work in IPS mode in any of these configurations with the new kernel? Anything else I could try to improve the situation?

[Oliver]

Diving a bit deeper, this is what is seems like:

• Current OPNsense kernels use drivers from the Realtek website. These are based on an old implementation of the FreeBSD re driver, which does not support netmap.
• When Suricata starts up in IPS mode and tries to use netmap, it fails with the Realtek drivers (while the standard FreeBSD re driver would provide the necessary netmap support).
• Even in non-IPS mode, Suricata would sometimes stop working until restarted. On some occasions, log messages "<Error> -- [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2" appeared in the log at very high frequency. In these cases, stopping Suricata took quite a while.
• My personal impression from observed stability problems and looking at the code is that the Realtek drivers have serious quality issues and should probably not be used in routers.

I thought I'd put this here for reference.