Suricata error, DNS crashes

[Anon87]

Hi!

First of all, thank you very much for your awesome work! It's been a wonderful ride so far.

I upgraded this morning to OPNsense 17.7, but ran across an issue. As soon as Suricata is enabled, the DNS seems to fail after a few minutes. Traffic is still possible, however, DNS queries fail. As soon as I disable Suricata, DNS queries succeed again. There are no rules set to block which might have an effect on this, the Suricata logs mentions doesn't report any block action whatsoever.

I tried it with the default supplied version, Suricata 3.2.3, upgraded to 4.0.0, but the problem remains. However, suricata.log (/var/log/suricata.log) does mention this everytime it's started:

Code Select Expand

[ERRCODE: SC_ERR_SYSCALL(50)] - Unable to set caps for iface "em0": Invalid argument

[franco]

Hi Anon,

I don't know where this would come from: Suricata and the FreeBSD version is the same. The only difference is that everything got reinstalled, which shouldn't cause this.

It could be that either the Suricata rules used changed during the fetch or that the new firewall rules generation is the problem, the first one being the more likely candidate.

Capability settings shouldn't fail, but it also shouldn't cause such a problem (only DNS).

Do you have gateway rules in the firewall, particularly about DNS?


Cheers,
Franco

[Anon87]

Hello Franco,

Thank you for your swift reply. If the Suricata rules did change and it led to some blocking, it should be listed on the alerts page, correct?

With the troubleshooting I did, one of the first things I thought about was the firewall ruleset. But it's pretty much the default, with all outgoing traffic allowed (IPv4) and the only incoming traffic is to the built in OpenVPN server, with I only enable if I leave home for more than one hour.

A few hours ago, I decided to completely reinstall the box, because theoretically, it could be a bug triggered by the configuration file, which I did import from 17.1.11. I began configuring it again, by hand. Walked through the setup wizard, selected the LibreSSL flavor, upgraded, configured the settings, enabled Suricata - worked all fine. Until I rebooted the machine, exact same problem as before.

I reinstalled it again, but now, I rebooted everytime I changed a single option. Countless reboots later, I think I've found the cause: with IPS mode enabled in Suricata, it causes the DNS queries/resolves to fail. After I disabled IPS mode, rebooted, Suricata ran fine, causing no problems.

Perhaps you can shine a light on this, as I am unfamiliar with the inner workings of Suricata. Is the IPS mode required to drop traffic, even if I set the default behaviour to drop in a specific rule or ruleset?

Last but not least, you mentioned that OPNsense 17.1 and 17.7 are based on the same FreeBSD version and run the same Suricata version. Could it be due to one of the mitigations/hardening techniques in HardenedBSD? As I read the releasenotes, there are some 'intense' changes made....

Thanks,
Anon87

Note to self: I should really, really get a KVM over IP solution for the headless systems at home

[franco]

Hi Anon87,

Well, if you talk about 17.1 and 17.7 the software versions differ greatly as both releases are 6 months apart. But every subsequent 17.1.x gets features from the 17.7 development branch over time. Due to this 17.1.11 and 17.7 are almost the same, except for a larger rework of the rule generation framework of the packet filter itself.

So if 17.1.11 was working fine for you, but 17.7 is giving you trouble now it means we can narrow the problem down to a handful of changes, none of which directly affect Suricata.

You should see alerts for drops, that's right. Also, is this a problem of IPv4 vs. IPv6 DNS resolution? I've seen Suricata drop IPv6 traffic but leave IPv4 alone?

Sorry this is a bit vague, it's not because we don't want to solve this, but because there is little indication of what could be wrong with the system. Need to pin this down more. :)


Cheers,
Franco

[Anon87]

Hi Franco,

Sure, I'll be glad to help out. I'll try to get into detail as much as possible, most will be completely worthless, but hope something can point you in the right direction.

I am able to reproduce this behaviour through the following steps:

• Perform a clean install of OPNsense - no configuration import whatsoever.
• After the reboot, open the web configurator and walk through the wizard. My system is directly connected through the internet, with a static, public IPv4 IP, part of a subnet. I configured this, with the /29 netmask . Didn't touch any of the other options. As a DNS provider, I use OpenDNS (208.67.220.220, 208.67.222.222).
• After the wizard, I've switched the flavour to LibreSSL and updated the corresponding packages.
• If I, right after that, disable the hardware acceleration (as mentioned in the 'IPS mode' helptext), enable all the rules (except for web-client, except for web-client, game, and inappropiate), download them and enable Suricata with the options from the first posted screenshot, reboot afterwards, it happens.
• If I disable Suricata or disable the IPS mode followed by a reboot, DNS resolves just fine.

My provider doesn't support IPv6 as of now, so that shouldn't be a problem. The only thing I haven't tried - yet - is to try another DNS provider. I am willing to downgrade to 17.1, to make 100% sure that this is 17.7 only behaviour. However, I might need a day or two to get that done, as I am away from home right now.

Cheers
J.

[Vincent]

Hello Anon87 and Franco,

I experienced similar issue with my upgrade path from 17.1.11 to 17.7
My config had suricata enabled (in IDS mode) and DNS activated.
All working fine in 17.1.11

After upgrading to 17.7 i got what seems a similar issue to what Anon87 described: unbound DNS was not resolving any more DNS.
Only exceptions were my local domain hosts entries resolved properly.

Stopping/disabling suricata didn't change to the issue.
But i found in "System > Settings > General" an option that seemed new to me:

Code Select Expand

"Allow DNS server list to be overridden by DHCP/PPP on WAN"
It was checked by default, and unticking it made my DNS magically resolving back external named (and resisting to reboot issue mention by Anon87).

Could it be a reason for you too that the change you makes don't resist for a reboot Anon87?

I hope that experience sharing will help.

Cheers,
Vincent

[franco]

Just quickly, there is a German user seeing the same issue: https://forum.opnsense.org/index.php?topic=5603.0

With the multiple reports and a bit of testing we should be able to pin this down further, but it really seems to be either Suricata IPS mode or an associated DNS ruleset.

We would try Suricata 3.2.2 to be sure it's either the binary or a newer ruleset...

[Anon87]

Could it be a reason for you too that the change you makes don't resist for a reboot Anon87?

Dear Vincent,

Thank you kindly for your reply, really appreciate it. Theoretically speaking, it shouldn't make a difference, as my WAN is static, there is no DHCP server running upstream. However, I'll be glad to put this to the test nevertheless.

Unfortunately, I can't try this right now, as there is a transfer going on which cannot be interrupted. By estimation, the transfer should be done late tomorrow or early friday, then I am free to reboot as often as I wish ;-)

Also, I assume that you mean IPS mode? The difference is that IDS is detection, IPS is blocking :)

Cheers,
J.

EDIT: Vincent, since you've managed to get it working, if you look through the /var/log/suricata.log, is there something like this reported:

Code Select Expand

[ERRCODE: SC_ERR_SYSCALL(50)] - Unable to set caps for iface "em0": Invalid argument

We would try Suricata 3.2.2 to be sure it's either the binary or a newer ruleset...

Thanks Franco!

The same behaviour happens on v3.2.3 (the default) and v4.0.0, so from my viewpoint, the ruleset causing this issue is more logical than the binary. Does it help if I supply a list of the enabled rules?

I've used Google Translate to help me out with the German, but it doesn't really make any sense - it gets translated rather bad. I am able to follow and interpret spoken German, but reading it is something completely different, gheh.

[franco]

Hi J,

Well, 3.2.3 may not be it, but 3.2.2 could be it, it's rather strange. We should exclude as much as possible.

This is the old amd64 package:

# pkg add -f https://pkg.opnsense.org/snapshots/suricata-3.2.2.txz

Listing the rulesets used will certainly help. If it's a ruleset it should also be easy to find the one responsible by deactivating one by one?


Cheers,
Franco

[ford]

Hi Guys,

iam this user from the german thread.
Does google translator struggels on bavarien? ;-)

OK, as Franco said, downgrad suricata to 3.2.2.
Probably a solution for a moment, cause my FW-log tells me the same as before, but DNS works _with_ IDS!

Small question:
I do not redirect on WAN.
Which interface redirects (in NAT) your traffic to proxy?
I am using the plugin below.
Do you use plugin "os-intrusion-detection-content-pt-open"?

greets
Rainer

[Vincent]

Hello J,

I've checked in /var/log/suricata.log, including history of past days, no track of that set caps error message.

Suricata is bound on igb1 for me, maybe the driver has an impact on the caps.

Cheers,
Vincent

[ford]

Hi,

after downgrade to 3.2.2 and up and down....
i spend a lot of time to read log files and traces.
Traces as well from my WAN, which is connected to my FritzBox.
There I found some strange pings and DNS requests... propably v6 once.

My idea was to set
Services
>Unbound DNS
>>General
[X] Register IPv6 link-local addresses in the DNS Resolver DHCP IPv6 Link-local
et Voilá, for me it works.

But meanwhile there are several lib updates via GUI update
and "Surprisecata" works as well w/o v6 local-link DNS regs in v3.2.3 ;-)

Same for you?

cheers
Rainer

[franco]

Rainer, do you have a SLAAC type setting for your clients?

So do you suggest setting or unsetting Register link-local for 3.2.3 (and 4.0.0)?

[Anon87]

An update for my situation:

It seems like the caps error in the Suricata log, was due to a change in the interface assignments. There is one onboard ethernet controller and one through PCI express. I just changed the WAN interface from the onboard controller to the PCI express one, and poof, no more caps errors.

DNS seems to be resolving now, with IPS mode on. However, it does take a while. I am doing further investigation today, as to where that may come from.

[Noctur]

I had been having to reboot about 1-2x per day to recover connection - all bars green but no connection. I thought that simply what must be done.

Following this thread, I tried  Suri 3.2.2 - same reboot requirement, then reloaded 3.2.3 - same. Saw post above and realized my WAN is on an internal NIC and LAN on a PCIe NIC. Switched WAN to another PCIe NIC yesterday and haven't had to reboot since. This all with IPS and about half of the rules on. Now trying with VPN on - which has been a past issue.