Home
Help
Search
Login
Register
OPNsense Forum
»
Archive
»
16.7 Legacy Series
»
Suricata crashes following upgrade to 16.7
« previous
next »
Print
Pages: [
1
]
2
Author
Topic: Suricata crashes following upgrade to 16.7 (Read 18705 times)
Manxmann
Newbie
Posts: 24
Karma: 0
Suricata crashes following upgrade to 16.7
«
on:
July 28, 2016, 01:17:55 pm »
Hi Folks,
My first post
Firstly many thanks to the devs for a fantastic firewall platform!
Ok my problem, I've been running 16.1.x for some time now under XenServer 6.5 and latterly 7.0 and its been working like a charm.
This morning I upgraded from 16.1.20 to 16.7, everything appeared to go well and the dashboard now reports:
OPNsense 16.7-amd64
FreeBSD 10.3-RELEASE-p5
OpenSSL 1.0.2h 3 May 2016
However I have a problem in that Suricata now no longer runs, crashing shortly after starting.
My config is as follows:
IDS is 'enabled' and IPS mode turned on.
I have one monitor interface defined 'WAN' which is a standard ethernet port with a static IP address.
Pattern matcher is AHO
As far as rules go I have the following rules enabled.
%YAML 1.1
---
rule-files:
- compromised.rules
- emerging-exploit.rules
- modbus-events.rules
- smtp-events.rules
- dns-events.rules
- emerging-malware.rules
- app-layer-events.rules
- OPNsense.rules
- emerging-pop3.rules
- emerging-scan.rules
- emerging-trojan.rules
- emerging-web_client.rules
- emerging-web_server.rules
- abuse.ch.sslblacklist.rules
- abuse.ch.sslipblacklist.rules
- abuse.ch.dyre_sslipblacklist.rules
- abuse.ch.feodotracker.rules
~
If I manually start suricata from the cmd line I get the following:
root@XEN-FW: # /usr/local/bin/suricata --netmap --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
28/7/2016 -- 11:37:09 - <Info> - Including configuration file installed_rules.yaml.
Illegal instruction (core dumped)
I have tried disabling all the user selectible rules with no success.
root@XEN-FW:/var/log # cat suricata.log
28/7/2016 -- 11:37:09 - <Notice> - This is Suricata version 3.1.1 RELEASE
28/7/2016 -- 11:37:16 - <Notice> - all 2 packet processing threads, 4 management threads initialized, engine started.
DMESG :
pid 80601 (suricata), uid 0: exited on signal 4 (core dumped)
125.869766 [ 798] generic_netmap_dtor Restored native NA 0
236.646486 [ 266] generic_find_num_desc called, in tx 1024 rx 1024
236.659997 [ 274] generic_find_num_queues called, in txq 0 rxq 0
236.673112 [ 798] generic_netmap_dtor Restored native NA 0
236.688968 [ 266] generic_find_num_desc called, in tx 1024 rx 1024
236.702597 [ 274] generic_find_num_queues called, in txq 0 rxq 0
236.716042 [ 798] generic_netmap_dtor Restored native NA 0
236.729174 [ 266] generic_find_num_desc called, in tx 1024 rx 1024
236.742635 [ 274] generic_find_num_queues called, in txq 0 rxq 0
Any help greatly appreciated
Logged
weust
Hero Member
Posts: 650
Karma: 57
Re: Suricata crashes following upgrade to 16.7
«
Reply #1 on:
July 28, 2016, 01:45:56 pm »
Would this apply to you?
https://forum.opnsense.org/index.php?topic=3430.0
Logged
Hobbyist at home, sysadmin at work. Sometimes the first is mixed with the second.
Manxmann
Newbie
Posts: 24
Karma: 0
Re: Suricata crashes following upgrade to 16.7
«
Reply #2 on:
July 28, 2016, 02:08:00 pm »
Certainly would
Logged
Julien
Hero Member
Posts: 666
Karma: 33
Re: Suricata crashes following upgrade to 16.7
«
Reply #3 on:
July 28, 2016, 04:36:51 pm »
have you got it fixed ?
Logged
OPNsense 23.1.7_3-amd64
FreeBSD 13.1-RELEASE-p7
OpenSSL 1.1.1t 7 Feb 2023
franco
Administrator
Hero Member
Posts: 17481
Karma: 1589
Re: Suricata crashes following upgrade to 16.7
«
Reply #4 on:
July 28, 2016, 11:52:28 pm »
Let's try to see if this was the new version 3.1.x or the OS itself. From the console, revert to the last repo state, replace "latest" (OpenSSL) with "libressl" (LibreSSL) if needed...
# opnsense-update -sn "16.1\/latest"
# pkg install -f suricata
# opnsense-update -sn "16.7\/latest"
It should downgrade Suricata to 3.0.2 and Libhtp to 0.5.20.
Cheers,
Franco
Logged
Manxmann
Newbie
Posts: 24
Karma: 0
Re: Suricata crashes following upgrade to 16.7
«
Reply #5 on:
July 29, 2016, 12:30:37 pm »
Ok, I've regressed Suricata back to 3.0.2 as per the 16.1.x release stream as suggested and also re-enabled IPS mode.
I'm just over an hour in and so far everything seems to be working fine.
root@XEN-FW:~ # suricata --build-info
This is Suricata version 3.0.2 RELEASE
Features: IPFW PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 NETMAP HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS
SIMD support: none
Atomic intrisics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032), C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.20, linked against LibHTP v0.5.20
Suricata Configuration:
AF_PACKET support: no
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: yes
Netmap support: yes
DAG enabled: no
Napatech enabled: no
Unix socket enabled: yes
Detection enabled: yes
libnss support: no
libnspr support: no
libjansson support: yes
hiredis support: no
Prelude support: no
PCRE jit: yes
LUA support: no
libluajit: no
libgeoip: yes
Non-bundled htp: yes
Old barnyard2 support: no
CUDA enabled: no
Hyperscan support: no
Suricatasc install: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Profiling enabled: no
Profiling locks enabled: no
Coccinelle / spatch: no
Generic build parameters:
Installation prefix: /usr/local
Configuration directory: /usr/local/etc/suricata/
Log directory: /var/log/suricata/
--prefix /usr/local
--sysconfdir /usr/local/etc
--localstatedir /var
Host: amd64-portbld-freebsd10.2
Compiler: cc (exec name) / clang (real)
GCC Protect enabled: yes
GCC march native enabled: no
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -O2 -pipe -fstack-protector -fno-strict-aliasing -DOS_FREEBSD
PCAP_CFLAGS
SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
Logged
Manxmann
Newbie
Posts: 24
Karma: 0
Re: Suricata crashes following upgrade to 16.7
«
Reply #6 on:
August 06, 2016, 01:16:11 pm »
Upgraded to 16.7.1 today, IDS/IPS still broken.
Platform - XenServer i.e XN nic's not Intel.
Logged
Manxmann
Newbie
Posts: 24
Karma: 0
Re: Suricata crashes following upgrade to 16.7
«
Reply #7 on:
August 25, 2016, 03:32:22 pm »
Hey Folks,
I've tracked through all the IPS broken discussions and still have no solution to my problem.
This morning I upgrade to 16.7.2 / 10.3.p7 and just as before Securicata borks in either IPS or IDS mode.
Just to confirm, the firewall is running in a VM on a Citrix XenServer 7.0 server.
FreeBSD _IS_ XEN aware and as such uses the correct XN driver and XenServer provides no mechanism to override this, see dmesg below:
root@XEN-FW:~ # dmesg | grep -i Xen
XEN: Hypervisor version 4.6 detected.
ACPI APIC Table: <Xen HVM>
xen_et0: <Xen PV Clock> on motherboard
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
acpi0: <Xen> on motherboard
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 30 at device 3.0 on pci0
xenstore0: <XenStore> on xenpci0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn1: <Virtual Network Interface> at device/vif/1 on xenbusb_front0
xn2: <Virtual Network Interface> at device/vif/2 on xenbusb_front0
xbd0: 102400MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xn3: <Virtual Network Interface> at device/vif/3 on xenbusb_front0
xn4: <Virtual Network Interface> at device/vif/4 on xenbusb_front0
xn5: <Virtual Network Interface> at device/vif/5 on xenbusb_front0
xn3: backend features:xn6: <Virtual Network Interface> at device/vif/6 on xenbusb_front0
xenbusb_back0: <Xen Backend Devices> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
root@XEN-FW:~ #
Any updates?
Logged
franco
Administrator
Hero Member
Posts: 17481
Karma: 1589
Re: Suricata crashes following upgrade to 16.7
«
Reply #8 on:
August 26, 2016, 05:56:14 pm »
Hi there,
Someone suggested these Xen adjustments a few months back:
nettool -K <vif name> tx off
nettool -K <xen bridge> tx off
Not sure if this will help. I do not have any Xen setups.
Cheers,
Franco
Logged
Manxmann
Newbie
Posts: 24
Karma: 0
Re: Suricata crashes following upgrade to 16.7
«
Reply #9 on:
August 27, 2016, 09:28:07 am »
Thanks Franco,
Unfortunately this doesn't solve 'this' problem. Xen has a rather annoying bug it would seem with TX offload and FreeBSD which results in a huge performance drop when a FreeBSD VM forwards packets between VIF's. Having encountered this problem previously I already have VIF/Bridge/Physical and VM Offload engines disabled.
As before prior to 16.7 Securicata worked without issue
Logged
Manxmann
Newbie
Posts: 24
Karma: 0
Re: Suricata crashes following upgrade to 16.7
«
Reply #10 on:
November 22, 2016, 09:14:51 pm »
just updated to 16.7.9 on 2 boxes. Same result Suricata will not run:
root@XEN-FW:/usr/local/etc/suricata # suricata -c /usr/local/etc/suricata/suricata.yaml -i xn0 -v
22/11/2016 -- 20:06:05 - <Warning> - [ERRCODE: SC_WARN_FASTER_CAPTURE_AVAILABLE(275)] - faster capture option is available: NETMAP (--netmap=xn0). Use --pcap=xn0 to suppress this warning
22/11/2016 -- 20:06:05 - <Info> - Including configuration file installed_rules.yaml.
Illegal instruction (core dumped)
root@XEN-FW:/usr/local/etc/suricata #
Log:
22/11/2016 -- 20:12:31 - <Notice> - This is Suricata version 3.1.3 RELEASE
22/11/2016 -- 20:12:31 - <Info> - CPUs/cores online: 2
22/11/2016 -- 20:12:31 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:31 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:31 - <Info> - No 'host-mode': suricata is in IDS mode, using default setting 'sniffer-only'
22/11/2016 -- 20:12:37 - <Info> - 22 rule files processed. 10282 rules successfully loaded, 0 rules failed
22/11/2016 -- 20:12:37 - <Info> - 10282 signatures processed. 46 are IP-only rules, 3312 are inspecting packet payload, 7864 inspect application layer, 102 are decoder event only
22/11/2016 -- 20:12:39 - <Info> - Threshold config parsed: 0 rule(s) found
22/11/2016 -- 20:12:39 - <Info> - eve-log output device (regular) initialized: eve.json
22/11/2016 -- 20:12:39 - <Info> - stats output device (regular) initialized: stats.log
22/11/2016 -- 20:12:39 - <Info> - Going to use 1 thread(s)
22/11/2016 -- 20:12:39 - <Info> - using interface xn0
22/11/2016 -- 20:12:39 - <Info> - Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
22/11/2016 -- 20:12:39 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - Set snaplen to 1524 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - Going to use 1 thread(s)
22/11/2016 -- 20:12:39 - <Info> - using interface xn0
22/11/2016 -- 20:12:39 - <Info> - Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
22/11/2016 -- 20:12:39 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - Set snaplen to 1524 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - RunModeIdsPcapWorkers initialised
22/11/2016 -- 20:12:39 - <Notice> - all 2 packet processing threads, 4 management threads initialized, engine started.
«
Last Edit: November 22, 2016, 09:24:20 pm by Manxmann
»
Logged
franco
Administrator
Hero Member
Posts: 17481
Karma: 1589
Re: Suricata crashes following upgrade to 16.7
«
Reply #11 on:
November 23, 2016, 09:02:14 am »
The culprit has been identified as Hyperscan requiring SSE3 processor support. Try this:
This is an amd64 install, older AMD hardware? What crypto are you using? (OpenSSL or LibreSSL)
I can provide working packages here. But in the long run we need to solve this with FreeBSD ports.
Logged
Manxmann
Newbie
Posts: 24
Karma: 0
Re: Suricata crashes following upgrade to 16.7
«
Reply #12 on:
December 13, 2016, 05:11:03 pm »
Sorry for the delay
Aha! progress excellent!
Yes it's an Opteron 2431 (Six Core) DL385 G6 server.
Crypto is currently OpenSSL
«
Last Edit: December 14, 2016, 12:39:56 am by Manxmann
»
Logged
franco
Administrator
Hero Member
Posts: 17481
Karma: 1589
Re: Suricata crashes following upgrade to 16.7
«
Reply #13 on:
December 13, 2016, 07:59:30 pm »
There is a test build without Hyperscan:
# pkg add -f
https://pkg.opnsense.org/snapshots/suricata-no-hs-3.2.txz
And to revert to the default one:
# pkg install -f suricata
Cheers,
Franco
Logged
Manxmann
Newbie
Posts: 24
Karma: 0
Re: Suricata crashes following upgrade to 16.7
«
Reply #14 on:
December 14, 2016, 12:45:08 am »
Thanks franco!
Surciata has now been up and running for over 1hour!
Now the big question, is this something that can be addressed going forward or would the removal of SSE3 support cause too many performance issues? I guess I'm asking is my ability to run Suricata on this host on borrowed time?
Logged
Print
Pages: [
1
]
2
« previous
next »
OPNsense Forum
»
Archive
»
16.7 Legacy Series
»
Suricata crashes following upgrade to 16.7