Some ET rulesets emtpy

[athurdent]

Hi there,

finally switched from pfSense to OPNsense 20.1 and I really like it

I'm using the telemetry rule set with the code from Deciso.
One problem though, I was wondering why Suricata does not catch ET CINS, ET DROP or ET COMPROMISED anymore like it did frequently on my pfSense Suricata.

It seems the respective rulesets are empty, just enabled and downloaded all as a test fo this. All the 58B sized are empty.
How do I fix this?

Edit: seems to be a problem with the telemetry plugin. If I uninstall that, the rules are not empty anymore.

Code: [Select]

root@OPNsense:/usr/local/etc/suricata/rules # ls -lah
total 27224
drwxr-x---  2 root  wheel   2.0K Feb  1 08:22 .
drwxr-xr-x  5 root  wheel   512B Feb  1 08:17 ..
-rw-r-----  1 root  wheel    98B Feb  1 08:20 OPNsense.rules
-rw-r-----  1 root  wheel   233K Feb  1 08:20 abuse.ch.feodotracker.rules
-rw-r-----  1 root  wheel   932K Feb  1 08:20 abuse.ch.sslblacklist.rules
-rw-r-----  1 root  wheel    16K Feb  1 08:20 abuse.ch.sslipblacklist.rules
-rw-r-----  1 root  wheel    11M Feb  1 08:20 abuse.ch.urlhaus.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 botcc.portgrouped.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 botcc.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:18 ciarmy.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 compromised.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 drop.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 dshield.rules
-rw-r-----  1 root  wheel   2.7K Feb  1 08:20 emerging-activex.rules
-rw-r-----  1 root  wheel    37K Feb  1 08:20 emerging-attack_response.rules
-rw-r-----  1 root  wheel    13K Feb  1 08:20 emerging-chat.rules
-rw-r-----  1 root  wheel   3.8M Feb  1 08:20 emerging-current_events.rules
-rw-r-----  1 root  wheel   139K Feb  1 08:20 emerging-deleted.rules
-rw-r-----  1 root  wheel   5.2K Feb  1 08:20 emerging-dns.rules
-rw-r-----  1 root  wheel    18K Feb  1 08:20 emerging-dos.rules
-rw-r-----  1 root  wheel   132K Feb  1 08:20 emerging-exploit.rules
-rw-r-----  1 root  wheel   2.9K Feb  1 08:20 emerging-ftp.rules
-rw-r-----  1 root  wheel   6.6K Feb  1 08:20 emerging-games.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 emerging-icmp.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 emerging-icmp_info.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 emerging-imap.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 emerging-inappropriate.rules
-rw-r-----  1 root  wheel   151K Feb  1 08:20 emerging-info.rules
-rw-r-----  1 root  wheel   606K Feb  1 08:20 emerging-malware.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 emerging-misc.rules
-rw-r-----  1 root  wheel   800K Feb  1 08:20 emerging-mobile_malware.rules
-rw-r-----  1 root  wheel   2.8K Feb  1 08:20 emerging-netbios.rules
-rw-r-----  1 root  wheel    26K Feb  1 08:20 emerging-p2p.rules
-rw-r-----  1 root  wheel   217K Feb  1 08:20 emerging-policy.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 emerging-pop3.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 emerging-rpc.rules
-rw-r-----  1 root  wheel   6.8K Feb  1 08:20 emerging-scada.rules
-rw-r-----  1 root  wheel    47K Feb  1 08:20 emerging-scan.rules
-rw-r-----  1 root  wheel   3.5K Feb  1 08:20 emerging-shellcode.rules
-rw-r-----  1 root  wheel   3.5K Feb  1 08:20 emerging-smtp.rules
-rw-r-----  1 root  wheel   4.0K Feb  1 08:20 emerging-snmp.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 emerging-sql.rules
-rw-r-----  1 root  wheel   3.4K Feb  1 08:20 emerging-telnet.rules
-rw-r-----  1 root  wheel    58B Feb  1 08:20 emerging-tftp.rules
-rw-r-----  1 root  wheel   6.7M Feb  1 08:20 emerging-trojan.rules
-rw-r-----  1 root  wheel    38K Feb  1 08:20 emerging-user_agents.rules
-rw-r-----  1 root  wheel   4.5K Feb  1 08:20 emerging-voip.rules
-rw-r-----  1 root  wheel    86K Feb  1 08:20 emerging-web_client.rules
-rw-r-----  1 root  wheel    36K Feb  1 08:20 emerging-web_server.rules
-rw-r-----  1 root  wheel    13K Feb  1 08:20 emerging-web_specific_apps.rules
-rw-r-----  1 root  wheel    10K Feb  1 08:20 emerging-worm.rules
-rw-r-----  1 root  wheel    23K Feb  1 08:20 opnsense.file_transfer.rules
-rw-r-----  1 root  wheel    15K Feb  1 08:20 opnsense.mail.rules
-rw-r-----  1 root  wheel    11K Feb  1 08:20 opnsense.media_streaming.rules
-rw-r-----  1 root  wheel    12K Feb  1 08:20 opnsense.messaging.rules
-rw-r-----  1 root  wheel    12K Feb  1 08:20 opnsense.social_media.rules
-rw-r-----  1 root  wheel   392B Feb  1 08:20 opnsense.test.rules
-rw-r-----  1 root  wheel   1.2K Feb  1 08:20 opnsense.uncategorized.rules
-rw-r-----  1 root  wheel   1.0M Feb  1 08:22 rules.sqlite
-rw-r-----  1 root  wheel     0B Feb  1 08:22 rules.sqlite.LCK
-rw-r-----  1 root  wheel   151K Feb  1 08:18 telemetry_sids.txt
-rw-r-----  1 root  wheel   113B Feb  1 08:18 telemetry_version.json
-rw-r-----  1 root  wheel    58B Feb  1 08:20 tor.rules

root@OPNsense:/usr/local/etc/suricata/rules # cat ciarmy.rules
#@opnsense_download_hash:4e3f6edde96c40618e17f846a****


[athurdent]

So, I found this: https://forum.opnsense.org/index.php?topic=12119.msg55567#msg55567
Can someone explain how that does make sense?

According to the changelogs here, the empty rules in OPNsense are are actively maintained in the ET pro set https://rules.emergingthreats.net/changelogs/

They are also the ones that catch the most offenders here, at least running the ET Telemetry Edition did not really catch much of those. How can the Telemetry ones be a good replacement then?

[opnip]

I asked me the same. Would be great if someone could explain why these rules are empty.

From the documentation, the categories exists:
https://tools.emergingthreats.net/docs/ETPro%20Rule%20Categories.pdf

Documentation old? ETPro Rulesets/Categories old? I'm a little bit confused about the ET telemetry option now.

[athurdent]

I'm using the ET open rules for now. Steadily blocking offenders with the categories missing from the Telemetry set.
I was under the impression that the Telemetry set would be good for all, users get pro rules for free, ET receives Telemetry. But if half the open rules are missing, why use that set?

[AdSchellevis]

The telemetry ruleset equals the pro ruleset minus some noisy rules not targeting actual threats (more false positives), if you miss a specific rule which you think should be in telemetry, we can always ask our friends at Proofpoint about it.
Just let us know, and we’ll pass the message.

The open ruleset doesn’t offer the same level of protection that ETpro(telemetry) does.

Best regards,

Ad

[athurdent]

Hi Ad,

many thanks for looking into this!

At least ET CINS and ET DROP seem to be actively maintained in the ET Pro set (changelog suggests that: https://rules.emergingthreats.net/changelogs/suricata-5.0-enhanced.etpro.2020-02-12T00:21:07.txt ).
And while noisy, those block offenders based on their current reputation, which seems to be a good thing, doesn't it?
Would be great if you could ask the nice people at Proofpoint to add those sets to the Telemetry set. 

Many thanks again!

[opnip]

From the level of protection in "etpro-telemetry" maybe the rules from botcc makes also sense.

They are updated regulary (changelog suggests that: https://rules.emergingthreats.net/changelogs/suricata-5.0-enhanced.etpro.2020-02-12T00:21:07.txt ):
ET CNC Shadowserver Reported CnC Server Port 80 Group 1 (botcc.portgrouped.rules)

Thx

[AdSchellevis]

Hi,

I've emailed a bit with Proofpoint, the rules in question are from ET-Open and are indeed less suited for the Telemetry product (false positives, mostly ip lists if I'm not mistaken).

In theory you should be able to apply both rulesets (pro telemetry + open) to Suricata, but since rules overlap it would lead to quite some (harmless) errors in the suricata log. We currently don't support both sets to be enabled at the same time, since it would be confusing for most of our users.

Short term, you could add a script which downloads and adds the requested rules, in /usr/local/etc/suricata/rules/telemetry_sids.txt you should be able to find all sids that are being delivered by ETPro-Telemetry.

If more people would want to use both rulesets, maybe its best to open a ticket on our end to request for support for both at the same time (https://github.com/opnsense/core/issues), which needs changes to core and another plugin to pull in the rules.

Best regards,

Ad

[athurdent]

Hi Ad,

great, many thanks for your efforts and insight!