OPNsense Forum
Archive => 17.1 Legacy Series => Topic started by: csmall on March 05, 2017, 07:07:21 pm
-
I would like to find out who is running 17.1.2 with IPS enabled and ET rules.
Is it working?
Do you have ET rules triggered and blocked?
Do you have any other rules enabled? Are they triggered and blocked?
If it working, what hardware are you running it on? NIC's etc...
I ask because it doesn't seem to work for me and another person I know running completely different hardware (better) with a fresh install of 17.1.2 has the exact same experience. Built in suricata rules trigger and custom geoip rules trigger, but that's it. None of the downloaded (ET) rules seem to work.
A fresh install of the latest pfsense using suricata on the same hardware results in ET rules triggering and blocking as expected.
I'm trying to figure out the cause and it would help to know what others are experiencing.
Thanks.
-
Hi csmall,
The easiest way to check your setup is to enable the "OPNsense/test rules" and choose as input filter "change all alerts to block", then download and enable suricata in IPS mode.
I just tried it on my machine (virtual / parallels), and it generates an alert "OPNsense test eicar virus" (blocked) when downloading the following test file:
http://www.eicar.org/download/eicar.com.txt
Best regards,
Ad
-
Thanks.
That works fine.
ET rules do not work.
Abuse rules do not work.
Geoip rules work, the test you just suggested works, the built in suricata rules work.
Franco compared the difference in configs between my working pfsense suricata with ET rules and my opnsense config and there was minimal difference.
I even tried making the changes to match but it changed nothing.
He did mention that the logging facility configuration was different but I'm not sure how much that matters.
I'm trying to figure out why ET and abuse rules do not work for me and at least one other person I know with completely different hardware .. both of us with clean installs of 17.1.2.
Are they working for anyone?
Thanks!
-
Ok, I'm not sure how you come to the conclusion that the rules don't work, the mechanism is the same for all.
Let's do some additional testing, go to the console and execute the following commands in sequence:
service suricata stop
/usr/local/bin/suricata -D -vvvv --netmap --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
grep signatures /var/log/suricata.log
(starts suricata with more logging output.)
Mine reports something like this:
6/3/2017 -- 08:48:07 - <Info> - 1130 signatures processed. 0 are IP-only rules, 1 are inspecting packet payload, 845 inspect application layer, 98 are decoder event only
-
Thanks Ad,
I say it is broken because there is no visual indication in the web GUI that there are ET or Abuse alerts being triggered/Blocked.
On pfsense using suricata with the same rules, they trigger and block all day long. Ipfire with snort and the same rules, they also trigger all day. I get 0 triggered/blocked ET rules in the opnsense web GUI.
I suppose it is possible that the rules are working and just not showing up in the GUI?
I will attempt your troubleshooting suggestion tonight and see what I get.
It is interesting to note that this is not just me. Another person I know with completely different hardware has the exact same experience.
I was hoping to find out who actually has it working as expected and see what could be a solution or identify a larger issue if there is one.
I will report back with my findings from the troubleshooting steps.
-
Thanks Ad,
I say it is broken because there is no visual indication in the web GUI that there are ET or Abuse alerts being triggered/Blocked.
On pfsense using suricata with the same rules, they trigger and block all day long. Ipfire with snort and the same rules, they also trigger all day. I get 0 triggered/blocked ET rules in the opnsense web GUI.
I suppose it is possible that the rules are working and just not showing up in the GUI?
I will attempt your troubleshooting suggestion tonight and see what I get.
It is interesting to note that this is not just me. Another person I know with completely different hardware has the exact same experience.
I was hoping to find out who actually has it working as expected and see what could be a solution or identify a larger issue if there is one.
I will report back with my findings from the troubleshooting steps.
My results.
6/3/2017 -- 20:56:06 - <Info> - 349 signatures processed. 75 are IP-only rules, 163 are inspecting packet payload, 171 inspect application layer, 0 are decoder event only
The rules I have enabled right now are the ones I know are constantly triggered on my connection when using pfsense or ipfire.
-
Which rules are active? does the number of signatures match the number of activated rules? and which alerts did you see before and how where they triggered?
Maybe you can dump some alerts from the other installs, the rules beneath it aren't very difficult to inspect.
-
Drop, Dshield, scan and she'll code.
Numbers seem to match.
-
csmall and me have been discussing this a bit prior to this post, I looks like our reporting front end choice (eve) may not report all results. At least that's the only thing I've come up with so far. What would be helpful:
Show rules / log entries that trigger in pfSense and inspect them more closely for whether or not they can appear in the eve logs or not.
The inline IPS mode there is exactly the same in configuration, a test for matching up a few settings didn't help so far.
So this boils down to: does it trigger rules, if not why. And if yes, do we actually see the results?
Cheers,
Franco
-
Thanks Franco.
Yes this sounds reasonable because everything seems to be configured properly.
So maybe the reporting front end isn't displaying these alerts for some reason.
If I can reinstall pfsense this weekend, what would you like to see from there to help troubleshoot this issue?
What could I grab exactly that would help?
Geoip and opnsense test rules show up. Abuse and ET do not
Thanks guys! This is my only issue with opnsense at the moment.
-
Is there anyway to see in realtime what suricata is blocking or what rules are triggered in a log file?
The suricata.log file doesn't show me any detail like that.
-
Exciting news Franco!
I had an ET rule trigger and block.
I was using IRC in the opnsense channel and I thought to myself, why not turn on the ET chat rule and see what happens.
The second I turned the rule on it triggered and dropped my irc connection .
It then proceeded to trigger other irc related alerts and blocks.
That is great! I still don't understand why no other ET rules show up but this is good news.
-
I wonder if I'm noticing that the majority of ET rules are not working because I have used multiple firewalls with ET rules all in a short period of time and know what to expect because of the results being identical in both of the other firewalls.
Others may not have exposure to other software using ET and just don't realize that they are not working right.
If I hadn't used pfsense and ipfire with ET I would just think opnsense isn't seeing anything that matches ET rules and that everything was normal.
Just a thought but I'm still convinced that something is wrong regarding ET rules/suricata in opnsense and I'd love to figure out what it is and get it resolved.
-
I really don't expect there's an issue there, given the fact that there are alerts triggered for some rules.
But if you test using another install, post the alerts that got triggered as I asked before, it should be quite easy to check what the underlaying logic is.
-
I can do that but it isn't easy to keep switching firewalls ya know? Maybe I can do it this weekend. :)
-
Hello everyone!
I have the exact same problem, I don't see any blocked alert in the web interface.
Did you make some progress on the issue?
Thank you!
-
With the risk of repeating myself, have you tried to install our test pattern and downloading the eicar test virus?
If the test rule functions, it's highly unlikely the other installed rules won't (the IRC rules from ET are also quite practical for testing purposes when there's IRC traffic).
-
With the risk of repeating myself, have you tried to install our test pattern and downloading the eicar test virus?
If the test rule functions, it's highly unlikely the other installed rules won't (the IRC rules from ET are also quite practical for testing purposes when there's IRC traffic).
Yeap! ;D
Sorry for not mentioning it right before!
Test scenario: clicked the link on eicar => loading forever, checked the alerts on opnsense, eicar alert present with block
-
I don't know what the problem is with ET rules in opnsense but they do not work right.
Or maybe pfsense with suricata and et rules is just full of tons of false positives all day and night.
And ipfire with snort and ET rules triggering the same rules.
I wish it worked in Opnsense, it is my only problem with opnsense. Everything else is so awesome.
The test opnsense rules work for me with the test virus, I managed to trigger 1 ET rules 1 time, the Chat one when accessing an IRC channel. Other than that, nada, crickets... just the built in suricata rules trigger.
-
Well as mentioned before, at our end all is working without issues, an occasional "ET POLICY Outbound Multiple Non-SMTP Server Emails" message on one of my test machines here also proving to me that ET rules do match.
@csmall, this is probably the last message I'm going to post here, I've requested concrete examples on multiple occasions and received none sofar. The engine works, if there's a difference in setup it would be very easy to find if we had concrete examples of rules to inspect.
-
I hear ya. I'm not the only one though. It isn't easy for me to install pfsense just to troubleshoot and switch back.
Don't take my posts as any sort of negative attack or anything like that. I love opnsense. The idps just doesn't appear to work properly/entirely for me and others.
I worked with Franco privately to try and discover differences between the configurations of pfsense and opnsense suricata implementations but the differences didn't jump out at him as anything that would be causing my issues.
Maybe it is working and just not reporting to the web interface or something? Others seem to experience the same problem.
I see the built in suricata rules show up in eve.json (I'm not sure if this is the correct place to watch for triggering of et rules or not)
When I see others post that they too experience what I am I feel like I should chime in as it is relevant.
Is there anything I can do other than install pfsense to try and help figure this out?
-
It's highly unlikely that the eve-log output in suricata is broken, but like stated before, there's really no way to help when there's no data to inspect.
It could very well be that another product installed some other rules which are more trigger friendly, but without knowing what is triggered for what reason on any other product there's really no way to tell if it's setup or something else.
Also an option is that another product is not only logging alerts, but other events as well, more information about the options of suricata can be found on their website http://suricata.readthedocs.io/en/latest/output/eve/eve-json-output.html (http://suricata.readthedocs.io/en/latest/output/eve/eve-json-output.html)
Originally we had more logging enabled (https://github.com/opnsense/core/blob/0cd3480d94e48cc80604ee825ff6cf43d618f542/src/opnsense/service/templates/OPNsense/IDS/suricata.yaml#L86-L109 (https://github.com/opnsense/core/blob/0cd3480d94e48cc80604ee825ff6cf43d618f542/src/opnsense/service/templates/OPNsense/IDS/suricata.yaml#L86-L109)), which we disabled later to avoid confusion.
In case you want to see more output, just test with all logging options on (edit /usr/local/etc/suricata/suricata.yaml) and restart suricata (service suricata restart).... which will provide you with quite some data.
-
Thanks. I'll check into these suggestions.
Hopefully one day I see something that makes it all make sense :)
-
I tried enabling the same logging from that link and no change.
Someday this will all make sense lol
Could this have anything to do with me using Realtek nics?
I doubt it but who knows. I have almost every ET rule enabled at this point and no alerts. I even tried scanning myself from the outside. The ET rules most commonly triggered on my connection through other products are the compromised rules, dshield rules, scan rules and shellcode rules.
Thanks for trying AdSchellevis. If anything else comes to mind that you think I can try to troubleshoot let me know. It is just difficult for me to swap firewall software back and forth for troubleshooting.
-
Now I have found out that there are some triggered ET rules, with ”blocked” action, as follows:
ET POLICY iTunes User Agent
ET POLICY PE EXE or DLL Windows file download HTTP
ET CHAT Skype User-Agent detected
ET DNS Query to a *.top domain - Likely Hostile
ET INFO HTTP Request to a *.pw domain
Well, rules trigger, and block. At least, the ones above did their job.
The question present here, among replies, still persists for me:
Are these rules the only rules that should have been triggered, and for the rest of rules, the internet was only clean enough? Or maybe are there attacks that OPNsense + Suricata don't detect? Cause using MS ForeFront/ ISA, and other paid IPS like Sophos XGFW & UTM does trigger allot more alerts, and blocks them!
As for what did I do so that at least these rules to be triggered, instead of none: I only disabled IDPS, reenabled it, reenabled rALL the rulesets, rescan and redownloaded the rulesets. I consider I did nothing different then before, but maybe somehow now is a difference. Still, I am not convinced that my action of disabling and reenabling the IDPS, the rulesets etc is the corrective measure. Time will tell, I plan on extending the use of OPNsense on two more branches, and on the main building soon. I'll come back with news, if any.
-
Thanks. I'll check into these suggestions.
Hopefully one day I see something that makes it all make sense :)
AdSchellevis
I broke down and installed pfsense. I'm seeing lots of et rules trigger.
What can I provide you to help troubleshoot why I don't have the same experience in opnsense?
Thanks!
-
Can you post some of the alerts including details? (if you replace ip addresses, please replace them for something similar for external/internal ranges)
-
Can you post some of the alerts including details? (if you replace ip addresses, please replace them for something similar for external/internal ranges)
Sure. I really hope we can figure this out.
I also made the move from a machine with Realtek nic to one with intel em*
I'll post some examples when I get home from work tonight.
-
Can you post some of the alerts including details? (if you replace ip addresses, please replace them for something similar for external/internal ranges)
AdSchellevis,
I PM'd you some examples. There are hundreds more as they fire all day and night. The same rules are triggered in Snort when using ipfire.
I get none of these in opnsense... it is quiet as a mouse.
-
I have enabled the IDS + IPS service on the WAN interface yesterday with the 'Aho-corasick'-patternmatcher. I then have enabled / downloaded a few rulesets, amongst which all the Abuse.ch ones, and the ET Malware, ET Mobile Malware and ET Exploit. All rulesets have been edited to 'drop' the packets. Tonight a first CRON run has been succesfully executed to download the newest versions of these rulesets.
This morning I am still able to download and open the EICAR test file and see no alert in my alerts.
I just tried it again with the opnsense-test-rules, but still nothing is blocked or alerted.
Please tell me what you need and I'll happily supply it to you.
Kr,
Vincent
-
Hi Vincent,
if the test rule doesn't work, you probably have other configuration issues, you best first try in IDS mode with "Promiscuous mode" enabled.
IPS doesn't work on all network drivers (needs solid netmap support).
Best regards,
Ad
@csmall as soon as I can find some time I will look at your logfiles and share my findings.
-
ok, I received the logs from csmall and as far as I can see the alerts reported are correctly suppressed (assuming that the wan ip isn't an internal reserved subnet).
Just one example out of the list:
05/15/2017-23:39:29.080055 [**] [1:2403332:3550] ET CINS Active Threat Intelligence Poor Reputation IP group 33 [**] [Classification: Misc Attack] [Priority: 2] {TCP} EXTERNALIP:38061 -> CSMALL_WANIP:2375
Defined by:
alert ip [<long list of addresses>] any -> $HOME_NET any (msg:"ET CINS Active Threat Intelligence Poor Reputation IP group 33"; reference:url,www.cinsscore.com; reference:url,www.networkcloaking.com/cins; threshold: type limit, track by_src, seconds 3600, count 1; classtype:misc-attack; sid:2403332; rev:3555;)
which should trigger when there's traffic coming from one of the external ip's in the IP reputation list and going to one of your internal networks.
Our internal networks are defined as:
HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]"
I guess someone thought it was a good idea to define the external wan addresses as internal networks, which will lead to quite some chatter.
more on suricata setup and recommended configurations https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Basic_Setup (https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Basic_Setup)
-
Great find Ad!
I wonder why pfsense and ipfire include the external IP's in the local subnet config.
I confirmed just now that my WAN IP is in fact listed in my HOME net config in pfsense.
After reading the link you sent, I don't understand why they have it set that way and what you are saying makes sense to me now.
So I assume if I added my WAN IP to the home subnet list in opnsense, I would see all these triggered as well, but they are likely just 'chatter' and not true intrusions.
Thank you for looking into this and I can't wait to get back on opnsense tonight.
-
I can confirm that you do see this 'chatter' when you add the WAN interface to HOME_NET, but these are real hits to the WAN side. I guess these 'hits' just never get pass the firewall, which is why you don't see them on the other interfaces. But then again, they are hits to the WAN and with IPS, you can block them at the source, so wouldn't that be better?
Since this is an older topic, I continued this conversation in thread
https://forum.opnsense.org/index.php?topic=6398.0
-
The firewall with no open ports and no pass rules will silently drop unsolicited incoming packets. In my opinion, that is usually best. Now, if I had SSH running on the WAN or other service installed in opensense that listened on the WAN, then there would be a great need to have IDS checking the WAN.