Suricata crashes following upgrade to 16.7

[Manxmann]

Hi Folks,

My first post 

Firstly many thanks to the devs for a fantastic firewall platform!

Ok my problem, I've been running 16.1.x for some time now under XenServer 6.5 and latterly 7.0 and its been working like a charm.

This morning I upgraded from 16.1.20 to 16.7, everything appeared to go well and the dashboard now reports:

OPNsense 16.7-amd64
FreeBSD 10.3-RELEASE-p5
OpenSSL 1.0.2h 3 May 2016

However I have a problem in that Suricata now no longer runs, crashing shortly after starting.

My config is as follows:

IDS is 'enabled' and IPS mode turned on.

I have one monitor interface defined 'WAN' which is a standard ethernet port with a static IP address.

Pattern matcher is AHO

As far as rules go I have the following rules enabled.

%YAML 1.1
---
rule-files:
 - compromised.rules
 - emerging-exploit.rules
 - modbus-events.rules
 - smtp-events.rules
 - dns-events.rules
 - emerging-malware.rules
 - app-layer-events.rules
 - OPNsense.rules
 - emerging-pop3.rules
 - emerging-scan.rules
 - emerging-trojan.rules
 - emerging-web_client.rules
 - emerging-web_server.rules
 - abuse.ch.sslblacklist.rules
 - abuse.ch.sslipblacklist.rules
 - abuse.ch.dyre_sslipblacklist.rules
 - abuse.ch.feodotracker.rules
~

If I manually start suricata from the cmd line I get the following:

root@XEN-FW: # /usr/local/bin/suricata --netmap --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
28/7/2016 -- 11:37:09 - <Info> - Including configuration file installed_rules.yaml.
Illegal instruction (core dumped)

I have tried disabling all the user selectible rules with no success.

root@XEN-FW:/var/log # cat suricata.log
28/7/2016 -- 11:37:09 - <Notice> - This is Suricata version 3.1.1 RELEASE
28/7/2016 -- 11:37:16 - <Notice> - all 2 packet processing threads, 4 management threads initialized, engine started.

DMESG :


pid 80601 (suricata), uid 0: exited on signal 4 (core dumped)
125.869766 [ 798] generic_netmap_dtor       Restored native NA 0
236.646486 [ 266] generic_find_num_desc     called, in tx 1024 rx 1024
236.659997 [ 274] generic_find_num_queues   called, in txq 0 rxq 0
236.673112 [ 798] generic_netmap_dtor       Restored native NA 0
236.688968 [ 266] generic_find_num_desc     called, in tx 1024 rx 1024
236.702597 [ 274] generic_find_num_queues   called, in txq 0 rxq 0
236.716042 [ 798] generic_netmap_dtor       Restored native NA 0
236.729174 [ 266] generic_find_num_desc     called, in tx 1024 rx 1024
236.742635 [ 274] generic_find_num_queues   called, in txq 0 rxq 0

Any help greatly appreciated

[weust]

Would this apply to you? https://forum.opnsense.org/index.php?topic=3430.0

[Manxmann]

Certainly would

[Julien]

have you got it fixed ?

[franco]

Let's try to see if this was the new version 3.1.x or the OS itself. From the console, revert to the last repo state, replace "latest" (OpenSSL) with "libressl" (LibreSSL) if needed...

# opnsense-update -sn "16.1\/latest"
# pkg install -f suricata
# opnsense-update -sn "16.7\/latest"

It should downgrade Suricata to 3.0.2 and Libhtp to 0.5.20.


Cheers,
Franco

[Manxmann]

Ok, I've regressed Suricata back to 3.0.2 as per the 16.1.x release stream as suggested and also re-enabled IPS mode.

I'm just over an hour in and so far everything seems to be working fine.

root@XEN-FW:~ # suricata --build-info
This is Suricata version 3.0.2 RELEASE
Features: IPFW PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 NETMAP HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS
SIMD support: none
Atomic intrisics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032), C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.20, linked against LibHTP v0.5.20

Suricata Configuration:
  AF_PACKET support:                       no
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            yes
  Netmap support:                          yes
  DAG enabled:                             no
  Napatech enabled:                        no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  libnss support:                          no
  libnspr support:                         no
  libjansson support:                      yes
  hiredis support:                         no
  Prelude support:                         no
  PCRE jit:                                yes
  LUA support:                             no
  libluajit:                               no
  libgeoip:                                yes
  Non-bundled htp:                         yes
  Old barnyard2 support:                   no
  CUDA enabled:                            no
  Hyperscan support:                       no

  Suricatasc install:                      no

  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no
  Profiling enabled:                       no
  Profiling locks enabled:                 no
  Coccinelle / spatch:                     no

Generic build parameters:
  Installation prefix:                     /usr/local
  Configuration directory:                 /usr/local/etc/suricata/
  Log directory:                           /var/log/suricata/

  --prefix                                 /usr/local
  --sysconfdir                             /usr/local/etc
  --localstatedir                          /var

  Host:                                    amd64-portbld-freebsd10.2
  Compiler:                                cc (exec name) / clang (real)
  GCC Protect enabled:                     yes
  GCC march native enabled:                no
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -O2 -pipe  -fstack-protector -fno-strict-aliasing -DOS_FREEBSD
  PCAP_CFLAGS
  SECCFLAGS                                -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security

[Manxmann]

Upgraded to 16.7.1 today, IDS/IPS still broken.

Platform - XenServer i.e XN nic's not Intel.

[Manxmann]

Hey Folks,

I've tracked through all the IPS broken discussions and still have no solution to my problem.

This morning I upgrade to 16.7.2 / 10.3.p7 and just as before Securicata borks in either IPS or IDS mode.

Just to confirm, the firewall is running in a VM on a Citrix XenServer 7.0 server.

FreeBSD _IS_ XEN aware and as such uses the correct XN driver and XenServer provides no mechanism to override this, see dmesg below:

root@XEN-FW:~ # dmesg | grep -i Xen
XEN: Hypervisor version 4.6 detected.
ACPI APIC Table: <Xen HVM>
xen_et0: <Xen PV Clock> on motherboard
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
acpi0: <Xen> on motherboard
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 30 at device 3.0 on pci0
xenstore0: <XenStore> on xenpci0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn1: <Virtual Network Interface> at device/vif/1 on xenbusb_front0
xn2: <Virtual Network Interface> at device/vif/2 on xenbusb_front0
xbd0: 102400MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xn3: <Virtual Network Interface> at device/vif/3 on xenbusb_front0
xn4: <Virtual Network Interface> at device/vif/4 on xenbusb_front0
xn5: <Virtual Network Interface> at device/vif/5 on xenbusb_front0
xn3: backend features:xn6: <Virtual Network Interface> at device/vif/6 on xenbusb_front0
xenbusb_back0: <Xen Backend Devices> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
root@XEN-FW:~ #

Any updates?

[franco]

Hi there,

Someone suggested these Xen adjustments a few months back:

nettool -K <vif name> tx off
nettool -K <xen bridge> tx off

Not sure if this will help. I do not have any Xen setups.


Cheers,
Franco

[Manxmann]

Thanks Franco,

Unfortunately this doesn't solve 'this' problem. Xen has a rather annoying bug it would seem with TX offload and FreeBSD which results in a huge performance drop when a FreeBSD VM forwards packets between VIF's. Having encountered this problem previously I already have VIF/Bridge/Physical and VM Offload engines disabled.

As before prior to 16.7 Securicata worked without issue
 

[Manxmann]

just updated to 16.7.9 on 2 boxes. Same result Suricata will not run:

root@XEN-FW:/usr/local/etc/suricata # suricata -c /usr/local/etc/suricata/suricata.yaml -i xn0 -v
22/11/2016 -- 20:06:05 - <Warning> - [ERRCODE: SC_WARN_FASTER_CAPTURE_AVAILABLE(275)] - faster capture option is available: NETMAP (--netmap=xn0). Use --pcap=xn0 to suppress this warning
22/11/2016 -- 20:06:05 - <Info> - Including configuration file installed_rules.yaml.
Illegal instruction (core dumped)
root@XEN-FW:/usr/local/etc/suricata #

Log:

22/11/2016 -- 20:12:31 - <Notice> - This is Suricata version 3.1.3 RELEASE
22/11/2016 -- 20:12:31 - <Info> - CPUs/cores online: 2
22/11/2016 -- 20:12:31 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:31 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:31 - <Info> - No 'host-mode': suricata is in IDS mode, using default setting 'sniffer-only'
22/11/2016 -- 20:12:37 - <Info> - 22 rule files processed. 10282 rules successfully loaded, 0 rules failed
22/11/2016 -- 20:12:37 - <Info> - 10282 signatures processed. 46 are IP-only rules, 3312 are inspecting packet payload, 7864 inspect application layer, 102 are decoder event only
22/11/2016 -- 20:12:39 - <Info> - Threshold config parsed: 0 rule(s) found
22/11/2016 -- 20:12:39 - <Info> - eve-log output device (regular) initialized: eve.json
22/11/2016 -- 20:12:39 - <Info> - stats output device (regular) initialized: stats.log
22/11/2016 -- 20:12:39 - <Info> - Going to use 1 thread(s)
22/11/2016 -- 20:12:39 - <Info> - using interface xn0
22/11/2016 -- 20:12:39 - <Info> - Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
22/11/2016 -- 20:12:39 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - Set snaplen to 1524 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - Going to use 1 thread(s)
22/11/2016 -- 20:12:39 - <Info> - using interface xn0
22/11/2016 -- 20:12:39 - <Info> - Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
22/11/2016 -- 20:12:39 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - Set snaplen to 1524 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - RunModeIdsPcapWorkers initialised
22/11/2016 -- 20:12:39 - <Notice> - all 2 packet processing threads, 4 management threads initialized, engine started.

[franco]

The culprit has been identified as Hyperscan requiring SSE3 processor support. Try this:

This is an amd64 install, older AMD hardware? What crypto are you using? (OpenSSL or LibreSSL)

I can provide working packages here. But in the long run we need to solve this with FreeBSD ports.

[Manxmann]

Sorry for the delay

Aha! progress excellent!

Yes it's an Opteron 2431 (Six Core) DL385 G6 server.

Crypto is currently OpenSSL

[franco]

There is a test build without Hyperscan:

# pkg add -f https://pkg.opnsense.org/snapshots/suricata-no-hs-3.2.txz

And to revert to the default one:

# pkg install -f suricata


Cheers,
Franco

[Manxmann]

Thanks franco!

Surciata has now been up and running for over 1hour!

Now the big question, is this something that can be addressed going forward or would the removal of SSE3 support cause too many performance issues? I guess I'm asking is my ability to run Suricata on this host on borrowed time?