OPNsense Forum
Archive => 16.7 Legacy Series => Topic started by: Manxmann on July 28, 2016, 01:17:55 pm
-
Hi Folks,
My first post :)
Firstly many thanks to the devs for a fantastic firewall platform!
Ok my problem, I've been running 16.1.x for some time now under XenServer 6.5 and latterly 7.0 and its been working like a charm.
This morning I upgraded from 16.1.20 to 16.7, everything appeared to go well and the dashboard now reports:
OPNsense 16.7-amd64
FreeBSD 10.3-RELEASE-p5
OpenSSL 1.0.2h 3 May 2016
However I have a problem in that Suricata now no longer runs, crashing shortly after starting.
My config is as follows:
IDS is 'enabled' and IPS mode turned on.
I have one monitor interface defined 'WAN' which is a standard ethernet port with a static IP address.
Pattern matcher is AHO
As far as rules go I have the following rules enabled.
%YAML 1.1
---
rule-files:
- compromised.rules
- emerging-exploit.rules
- modbus-events.rules
- smtp-events.rules
- dns-events.rules
- emerging-malware.rules
- app-layer-events.rules
- OPNsense.rules
- emerging-pop3.rules
- emerging-scan.rules
- emerging-trojan.rules
- emerging-web_client.rules
- emerging-web_server.rules
- abuse.ch.sslblacklist.rules
- abuse.ch.sslipblacklist.rules
- abuse.ch.dyre_sslipblacklist.rules
- abuse.ch.feodotracker.rules
~
If I manually start suricata from the cmd line I get the following:
root@XEN-FW: # /usr/local/bin/suricata --netmap --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
28/7/2016 -- 11:37:09 - <Info> - Including configuration file installed_rules.yaml.
Illegal instruction (core dumped)
I have tried disabling all the user selectible rules with no success.
root@XEN-FW:/var/log # cat suricata.log
28/7/2016 -- 11:37:09 - <Notice> - This is Suricata version 3.1.1 RELEASE
28/7/2016 -- 11:37:16 - <Notice> - all 2 packet processing threads, 4 management threads initialized, engine started.
DMESG :
pid 80601 (suricata), uid 0: exited on signal 4 (core dumped)
125.869766 [ 798] generic_netmap_dtor Restored native NA 0
236.646486 [ 266] generic_find_num_desc called, in tx 1024 rx 1024
236.659997 [ 274] generic_find_num_queues called, in txq 0 rxq 0
236.673112 [ 798] generic_netmap_dtor Restored native NA 0
236.688968 [ 266] generic_find_num_desc called, in tx 1024 rx 1024
236.702597 [ 274] generic_find_num_queues called, in txq 0 rxq 0
236.716042 [ 798] generic_netmap_dtor Restored native NA 0
236.729174 [ 266] generic_find_num_desc called, in tx 1024 rx 1024
236.742635 [ 274] generic_find_num_queues called, in txq 0 rxq 0
Any help greatly appreciated :)
-
Would this apply to you? https://forum.opnsense.org/index.php?topic=3430.0
-
Certainly would :)
-
have you got it fixed ?
-
Let's try to see if this was the new version 3.1.x or the OS itself. From the console, revert to the last repo state, replace "latest" (OpenSSL) with "libressl" (LibreSSL) if needed...
# opnsense-update -sn "16.1\/latest"
# pkg install -f suricata
# opnsense-update -sn "16.7\/latest"
It should downgrade Suricata to 3.0.2 and Libhtp to 0.5.20.
Cheers,
Franco
-
Ok, I've regressed Suricata back to 3.0.2 as per the 16.1.x release stream as suggested and also re-enabled IPS mode.
I'm just over an hour in and so far everything seems to be working fine.
root@XEN-FW:~ # suricata --build-info
This is Suricata version 3.0.2 RELEASE
Features: IPFW PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 NETMAP HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_LIBJANSSON TLS
SIMD support: none
Atomic intrisics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032), C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.20, linked against LibHTP v0.5.20
Suricata Configuration:
AF_PACKET support: no
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: yes
Netmap support: yes
DAG enabled: no
Napatech enabled: no
Unix socket enabled: yes
Detection enabled: yes
libnss support: no
libnspr support: no
libjansson support: yes
hiredis support: no
Prelude support: no
PCRE jit: yes
LUA support: no
libluajit: no
libgeoip: yes
Non-bundled htp: yes
Old barnyard2 support: no
CUDA enabled: no
Hyperscan support: no
Suricatasc install: no
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Profiling enabled: no
Profiling locks enabled: no
Coccinelle / spatch: no
Generic build parameters:
Installation prefix: /usr/local
Configuration directory: /usr/local/etc/suricata/
Log directory: /var/log/suricata/
--prefix /usr/local
--sysconfdir /usr/local/etc
--localstatedir /var
Host: amd64-portbld-freebsd10.2
Compiler: cc (exec name) / clang (real)
GCC Protect enabled: yes
GCC march native enabled: no
GCC Profile enabled: no
Position Independent Executable enabled: no
CFLAGS -O2 -pipe -fstack-protector -fno-strict-aliasing -DOS_FREEBSD
PCAP_CFLAGS
SECCFLAGS -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
-
Upgraded to 16.7.1 today, IDS/IPS still broken.
Platform - XenServer i.e XN nic's not Intel.
-
Hey Folks,
I've tracked through all the IPS broken discussions and still have no solution to my problem.
This morning I upgrade to 16.7.2 / 10.3.p7 and just as before Securicata borks in either IPS or IDS mode.
Just to confirm, the firewall is running in a VM on a Citrix XenServer 7.0 server.
FreeBSD _IS_ XEN aware and as such uses the correct XN driver and XenServer provides no mechanism to override this, see dmesg below:
root@XEN-FW:~ # dmesg | grep -i Xen
XEN: Hypervisor version 4.6 detected.
ACPI APIC Table: <Xen HVM>
xen_et0: <Xen PV Clock> on motherboard
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
acpi0: <Xen> on motherboard
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 30 at device 3.0 on pci0
xenstore0: <XenStore> on xenpci0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn1: <Virtual Network Interface> at device/vif/1 on xenbusb_front0
xn2: <Virtual Network Interface> at device/vif/2 on xenbusb_front0
xbd0: 102400MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xn3: <Virtual Network Interface> at device/vif/3 on xenbusb_front0
xn4: <Virtual Network Interface> at device/vif/4 on xenbusb_front0
xn5: <Virtual Network Interface> at device/vif/5 on xenbusb_front0
xn3: backend features:xn6: <Virtual Network Interface> at device/vif/6 on xenbusb_front0
xenbusb_back0: <Xen Backend Devices> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
root@XEN-FW:~ #
Any updates?
-
Hi there,
Someone suggested these Xen adjustments a few months back:
nettool -K <vif name> tx off
nettool -K <xen bridge> tx off
Not sure if this will help. I do not have any Xen setups.
Cheers,
Franco
-
Thanks Franco,
Unfortunately this doesn't solve 'this' problem. Xen has a rather annoying bug it would seem with TX offload and FreeBSD which results in a huge performance drop when a FreeBSD VM forwards packets between VIF's. Having encountered this problem previously I already have VIF/Bridge/Physical and VM Offload engines disabled.
As before prior to 16.7 Securicata worked without issue :(
-
just updated to 16.7.9 on 2 boxes. Same result Suricata will not run:
root@XEN-FW:/usr/local/etc/suricata # suricata -c /usr/local/etc/suricata/suricata.yaml -i xn0 -v
22/11/2016 -- 20:06:05 - <Warning> - [ERRCODE: SC_WARN_FASTER_CAPTURE_AVAILABLE(275)] - faster capture option is available: NETMAP (--netmap=xn0). Use --pcap=xn0 to suppress this warning
22/11/2016 -- 20:06:05 - <Info> - Including configuration file installed_rules.yaml.
Illegal instruction (core dumped)
root@XEN-FW:/usr/local/etc/suricata #
Log:
22/11/2016 -- 20:12:31 - <Notice> - This is Suricata version 3.1.3 RELEASE
22/11/2016 -- 20:12:31 - <Info> - CPUs/cores online: 2
22/11/2016 -- 20:12:31 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:31 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:31 - <Info> - No 'host-mode': suricata is in IDS mode, using default setting 'sniffer-only'
22/11/2016 -- 20:12:37 - <Info> - 22 rule files processed. 10282 rules successfully loaded, 0 rules failed
22/11/2016 -- 20:12:37 - <Info> - 10282 signatures processed. 46 are IP-only rules, 3312 are inspecting packet payload, 7864 inspect application layer, 102 are decoder event only
22/11/2016 -- 20:12:39 - <Info> - Threshold config parsed: 0 rule(s) found
22/11/2016 -- 20:12:39 - <Info> - eve-log output device (regular) initialized: eve.json
22/11/2016 -- 20:12:39 - <Info> - stats output device (regular) initialized: stats.log
22/11/2016 -- 20:12:39 - <Info> - Going to use 1 thread(s)
22/11/2016 -- 20:12:39 - <Info> - using interface xn0
22/11/2016 -- 20:12:39 - <Info> - Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
22/11/2016 -- 20:12:39 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - Set snaplen to 1524 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - Going to use 1 thread(s)
22/11/2016 -- 20:12:39 - <Info> - using interface xn0
22/11/2016 -- 20:12:39 - <Info> - Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
22/11/2016 -- 20:12:39 - <Info> - Found an MTU of 1500 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - Set snaplen to 1524 for 'xn0'
22/11/2016 -- 20:12:39 - <Info> - RunModeIdsPcapWorkers initialised
22/11/2016 -- 20:12:39 - <Notice> - all 2 packet processing threads, 4 management threads initialized, engine started.
-
The culprit has been identified as Hyperscan requiring SSE3 processor support. Try this:
This is an amd64 install, older AMD hardware? What crypto are you using? (OpenSSL or LibreSSL)
I can provide working packages here. But in the long run we need to solve this with FreeBSD ports.
-
Sorry for the delay
Aha! progress excellent!
Yes it's an Opteron 2431 (Six Core) DL385 G6 server.
Crypto is currently OpenSSL
-
There is a test build without Hyperscan:
# pkg add -f https://pkg.opnsense.org/snapshots/suricata-no-hs-3.2.txz
And to revert to the default one:
# pkg install -f suricata
Cheers,
Franco
-
Thanks franco!
Surciata has now been up and running for over 1hour!
Now the big question, is this something that can be addressed going forward or would the removal of SSE3 support cause too many performance issues? I guess I'm asking is my ability to run Suricata on this host on borrowed time?
-
FreeBSD never went with the proposed Hyperscan inclusion, that's why it's not a pressing issue there.
The core problem is that Hyperscan requires SSE3 and has no fallbacks without it.
That makes full amd64 compatibility impossible.
I have an idea how to provide support for both, but without FreeBSD requiring this change it will be upon us to introduce it. I don't have an ETA.
There are two workarounds:
1. Use i386 for your hardware. That's a bit of a pain to redo the install.
2. Use the package provided here and "lock" it from the Firmware GUI packages list so that it won't be overwritten on updates. Whenever a new Suricata comes out let us know and we will provide a newer package.
Cheers,
Franco
-
Upon further discussion the CPU should have SSE3, but we've previously seen issues with AMD in general with regard to Hyperscan. We will be keeping a close watch on the issue in any case.
-
Thanks franco,
That sounds like an interesting situation. As per your recommendation I will lock the Suricata package now from updates and call on your generosity should a major update occur that I need to deploy.
With regard to your latter statement yes according to everything I can find it should support SSE3 however looking at a Linux VM running on the same host I get the following, as you can see SSE3 is missing for some reason. I've checked on a couple of DL385G6 servers are the results are the same, I'll start looking to see if I can find any microcode issues/updates on Google:
/proc# cat cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 8
model name : Six-Core AMD Opteron(tm) Processor 2431
stepping : 0
microcode : 0x10000da
cpu MHz : 2400.160
cache size : 512 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow rep_good nopl extd_apicid pni cx16 x2apic popcnt hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs vmmcall
bogomips : 4800.32
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management:
-
Silly me, the Opteron Istanbul core does support SSE3 and is shown in the DMESG of OPNSense booting however it doesn't support SSSE3 (extra S). Could this be what HYperscan is using not SSE3?
CPU: Six-Core AMD Opteron(tm) Processor 2431 (2400.14-MHz K8-class CPU)
Origin="AuthenticAMD" Id=0x100f80 Family=0x10 Model=0x8 Stepping=0
Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x80a02001<SSE3,CX16,x2APIC,POPCNT,HV>
AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
AMD Features2=0x5f3<LAHF,CMP,CR8,ABM,SSE4A,MAS,Prefetch,IBS>
-
You are right. The minimum requirement for Hyperscan is "core2", which added SSSE3 via Intel.
https://github.com/01org/hyperscan/issues/20#issuecomment-218335994
What I don't get is why instruction sets are run during load-time, which lockes out everyone not capable even though Hyperscan wasn't even selected.
Cheers,
Franco
-
First, I want to thanks the OPNsense team for such great and very responsive firewall software, very happy user here.
Wanted to post in this thread since already related to the OP issue and could be informative to some with similar legacy AMD hardware.
After running OPNsense 16.7.x for a while with outstanding performance results for my relatively old hardware (AMD Athlon 64 X2 5600+, 4GB DDR2, IDE HDD) at home, I decided to start using IPS and followed documentation and only enabled all the "abuse.ch" to start being familiarized with, they were working almost for a week with lots of alerts, until I started reading more about and enabled some "Emerging Threats", after applying the selected ET the service(suricata) started crashing withing seconds after apply and/or restarts.
After struggling with the logs and the "exited on signal 4 (core dumped)" almost for a day, I found this thread and followed post #13 and locked the package from being overwritten as previously denoted and my frustration just ended with a smile, keep up the good work.
Best regards