OPNsense Forum

Archive => 24.1, 24.4 Legacy Series => Topic started by: Frickey on January 30, 2024, 04:50:01 PM

Title: 24.1 IDS breaks internet
Post by: Frickey on January 30, 2024, 04:50:01 PM
I just upgraded to 24.1 without changing any settings.
After the update when suricata service is running all the internet freezes. Hardware offloading is disabled, even re-enabling and disabling it doesnt work.

Any ideas where the problem might be?

Thanks.
Title: Re: 24.1 IDS breaks internet
Post by: danderson on January 30, 2024, 05:05:38 PM
maybe the same as it was ~6 months back when i was testing Suricata7.

https://forum.opnsense.org/index.php?topic=34997.msg
https://forum.opnsense.org/index.php?topic=35130.msg
Title: Re: 24.1 IDS breaks internet
Post by: franco on January 30, 2024, 05:42:28 PM
IPS mode I guess? Same same, but different every time. These things are hard to trace up front.


Cheers,
Franco
Title: Re: 24.1 IDS breaks internet
Post by: Frickey on January 30, 2024, 06:15:08 PM
it even happens when only ids is enabled and no ips. I'll try the configs from danderson
Title: Re: 24.1 IDS breaks internet
Post by: db9 on January 30, 2024, 06:20:07 PM
I have the same issue after upgrading to 24.1.

The Egress connection to Internet work for a couple of minutes when starting the firewall. After this period the traffic to outside stops. After disabling IPS (suricata) the connections are restored. In my case IPS is enabled on the WAN interface.

I have changed my custom file with the help with this post. Now it looks stable for a couple of minutes.

https://forum.opnsense.org/index.php?topic=35130.msg
Title: Re: 24.1 IDS breaks internet
Post by: seed on January 30, 2024, 06:36:47 PM
I must report the same issues.
Having suricata running breaks the connection.

When connected to the opnsense console i can ping 1.1 through the igb interface. But not to lan (lacp lagg with ixl interfaces)
Title: Re: 24.1 IDS breaks internet
Post by: seed on January 30, 2024, 06:46:34 PM
Quote from: seed on January 30, 2024, 06:36:47 PM
I must report the same issues.
Having suricata running breaks the connection.

When connected to the opnsense console i can ping 1.1 through the igb interface. But not to lan (lacp lagg with ixl interfaces)

Adding:

stream.midstream-policy: ignore
http2:
  enabled: yes
quic:
  enabled: yes


to /usr/local/opnsense/service/templates/OPNsense/IDS/custom.yaml fixed the issue as described in the upper post.
Looks like a little hotfix must be released.
Title: Re: 24.1 IDS breaks internet
Post by: fadern on January 30, 2024, 07:07:32 PM
Same issue here.
Running IPS on the LAN side.
Web-gui get unresponsive after a few minutes and the network works as crap.
Igb interfaces.
Title: Re: 24.1 IDS breaks internet
Post by: seed on January 30, 2024, 07:42:46 PM
Quote from: seed on January 30, 2024, 06:46:34 PM
Quote from: seed on January 30, 2024, 06:36:47 PM
I must report the same issues.
Having suricata running breaks the connection.

When connected to the opnsense console i can ping 1.1 through the igb interface. But not to lan (lacp lagg with ixl interfaces)

Adding:

stream.midstream-policy: ignore
http2:
  enabled: yes
quic:
  enabled: yes


to /usr/local/opnsense/service/templates/OPNsense/IDS/custom.yaml fixed the issue as described in the upper post.
Looks like a little hotfix must be released.


Even with the fix applied i have problems reaching my servers by http/https.
I disabled suricata for now.
Title: Re: 24.1 IDS breaks internet
Post by: vabello on January 30, 2024, 07:47:17 PM
Same issue. I have a very new installation from a few days ago. Suricata was enabled in IPS mode. I had only one rule set downloading for testing. After the upgrade, it seemed like most TCP traffic wasn't working through the firewall. DNS resolution with unbound was working fine. Echo requests/replies were working fine. I could load some things, but definitely not most. On a hunch, I eventually stopped Suricata and everything started working. I've just disabled it for now until I know what's going on.
Title: Re: 24.1 IDS breaks internet
Post by: saintjimmy on January 30, 2024, 08:29:21 PM
Same issue here, disabling Suricata works. It was on IPS on WAN interface.
What is weird is that the same issue occurs with Crowdsec.
Title: Re: 24.1 IDS breaks internet
Post by: peter.vynck on January 30, 2024, 08:37:36 PM
Same issue here but strangely enough only on 1 of the 2 WAN connections?!

The 'standard' WAN interface (igb0) stopped working but the other fiber interface (pppoe0) continued working. Both interfaces are in my Suricata interfaces list...
Title: Re: 24.1 IDS breaks internet
Post by: nomad49 on January 30, 2024, 09:06:49 PM
Hello,
same issue here.

after disabling suricata everything is stable again.

as soon as suricata is enabled the web interface freezes about 4 - 5 minutes later, no traffic goes through.

Versions    OPNsense 24.1-amd64
Protectli FW4C
Title: Re: 24.1 IDS breaks internet
Post by: Cerberus on January 30, 2024, 09:21:00 PM
For me, disabling Surricata is not enough. IPv4 WAN is complety dead, IPv6 still works. Unbound cant resolve anything but has IPv4 and IPv6 upstream servers.

Update: the system has no IPv4 default gateway anymore.
Title: Re: 24.1 IDS breaks internet
Post by: danderson on January 30, 2024, 10:14:08 PM
I have the same issue, even with the addition to the custom.yaml. Disabled suricata for now and all working.

Quote from: seed on January 30, 2024, 07:42:46 PM
Quote from: seed on January 30, 2024, 06:46:34 PM
Quote from: seed on January 30, 2024, 06:36:47 PM
I must report the same issues.
Having suricata running breaks the connection.

When connected to the opnsense console i can ping 1.1 through the igb interface. But not to lan (lacp lagg with ixl interfaces)

Adding:

stream.midstream-policy: ignore
http2:
  enabled: yes
quic:
  enabled: yes


to /usr/local/opnsense/service/templates/OPNsense/IDS/custom.yaml fixed the issue as described in the upper post.
Looks like a little hotfix must be released.


Even with the fix applied i have problems reaching my servers by http/https.
I disabled suricata for now.
Title: Re: 24.1 IDS breaks internet
Post by: franco on January 30, 2024, 10:15:26 PM
I've spread this message out over our communication channels:

Suricata 7 appears to have severe issues with Netmap mode, alerting is likely affected. We'll be reverting back to Suricata 6 tomorrow and recommend disabling IPS mode on 24.1 for now. Best done prior to executing the upgrade!


Cheers,
Franco
Title: Re: 24.1 IDS breaks internet
Post by: Sparkey on January 31, 2024, 12:09:16 AM
I have the same issue after upgrading to 24.1.   Disable all is okay.
Sparkey
Title: Re: 24.1 IDS breaks internet
Post by: daygle on January 31, 2024, 12:52:15 AM
Same issue with me.

Disabling IDS OPNsense started working again. Hope a fix is developed soon.
Title: Re: 24.1 IDS breaks internet
Post by: rudiservo on January 31, 2024, 01:17:51 AM
Same here, had to disable it.

Going out on a limb here franco, sorry, I know you are the maintainer of the package, is it compiled with --enable-netmap?

I don't see it in Makefile of the master branch.

https://docs.suricata.io/en/suricata-7.0.2/capture-hardware/netmap.html

"To build Suricata with NETMAP, add --enable-netmap to the configure line. The location of the NETMAP includes (/usr/src/sys/net/) does not have to be specified."
Title: Re: 24.1 IDS breaks internet
Post by: rizzeaux on January 31, 2024, 03:25:47 AM
The fix worked for me with no issues but only after rebooting.

Quote from: seed on January 30, 2024, 07:42:46 PM
Quote from: seed on January 30, 2024, 06:46:34 PM
Quote from: seed on January 30, 2024, 06:36:47 PM
I must report the same issues.
Having suricata running breaks the connection.

When connected to the opnsense console i can ping 1.1 through the igb interface. But not to lan (lacp lagg with ixl interfaces)

Adding:

stream.midstream-policy: ignore
http2:
  enabled: yes
quic:
  enabled: yes


to /usr/local/opnsense/service/templates/OPNsense/IDS/custom.yaml fixed the issue as described in the upper post.
Looks like a little hotfix must be released.


Even with the fix applied i have problems reaching my servers by http/https.
I disabled suricata for now.
Title: Re: 24.1 IDS breaks internet
Post by: TParker31 on January 31, 2024, 04:01:28 AM
Thank you for pointing out this issue! Just wasted the last hour ripping apart my config and disabling every setting except IDS....
Title: Re: 24.1 IDS breaks internet
Post by: awptechnologies on January 31, 2024, 06:41:55 AM
Same Here will be waiting for fix and checking on forum. Gonna do testing and make sure it is fixed with update then revert to snapshot before update and redo update when fixed to ensure everything goes smooth with updating to 24.
Title: Re: 24.1 IDS breaks internet
Post by: Stephan81 on January 31, 2024, 07:34:38 AM
I've tried the solution mentioned in this thread, this doesn't resolve the issue.

Only working solution is to disable IPS option, Intrustion detection can remain Enabled (basically you know if something got in, but you didn't block it)

Tried the following without luck
- ET removal = nok
- removing all rules = nok
- reinstalling suricata = nok
- delayed start = nok
- removed internet WAN from blocking = nok (so IPS was only working on server WAN ip, all client internet traffic was unblocked/monitored)
- the fix mentioned in this thread

a hotfix with a downgrade, back to Suricata 6 seems the way to go.
Go TEAM OPNsense!

Title: Re: 24.1 IDS breaks internet
Post by: fadern on January 31, 2024, 09:23:13 AM
I have two boxes running nearly the same config. IPS is enabled on both boxes.
One are suffering from this issue and the other one is running fine. Both are based on Intel.
Title: Re: 24.1 IDS breaks internet
Post by: franco on January 31, 2024, 12:54:40 PM
Meanwhile Suricata has been rolled back from 7 to 6 anyway. Making broad statements with ambiguous context doesn't help.


Cheers,
Franco
Title: Re: 24.1 IDS breaks internet
Post by: seed on January 31, 2024, 01:14:14 PM
I hope it isn't postponed to somewhere in six months. Without any logs on hand it seems difficult to open a bugreport in the suricata github.

Edit: i meant the release of suricata 7. not the release of the rollback.
Title: Re: 24.1 IDS breaks internet
Post by: SLIM on January 31, 2024, 01:16:41 PM
Version 24.1_1 fixed IPS once I did a reboot. Thank you for the rollback
Title: Re: 24.1 IDS breaks internet
Post by: franco on January 31, 2024, 02:59:04 PM
@seedL: the development version still has Suricata 7. It has had it for a year now. Reports and problems have been very sparse so far. Actually, we don't know if it got worse somwhere between 7 RC1 where we started testing it, but it's not an immediate priority after the rollback. We will pick this up next week and see.


Cheers,
Franco
Title: Re: 24.1 IDS breaks internet
Post by: sdjme on January 31, 2024, 04:50:42 PM
Quote from: franco on January 31, 2024, 12:54:40 PM
Meanwhile Suricata has been rolled back from 7 to 6 anyway.

I never had Suricata installed, but it seems that 24.1_1 forced the package to install. Was this intended behavior?
Title: Re: 24.1 IDS breaks internet
Post by: Starrbuck on January 31, 2024, 05:05:30 PM
Quote from: sdjme on January 31, 2024, 04:50:42 PM
I never had Suricata installed, but it seems that 24.1_1 forced the package to install. Was this intended behavior?

Suricata is there as part of the base install.  Services > Intrusion Detection.
Title: Re: 24.1 IDS breaks internet
Post by: franco on January 31, 2024, 05:29:55 PM
Quote from: sdjme on January 31, 2024, 04:50:42 PM
Quote from: franco on January 31, 2024, 12:54:40 PM
Meanwhile Suricata has been rolled back from 7 to 6 anyway.

I never had Suricata installed, but it seems that 24.1_1 forced the package to install. Was this intended behavior?

If you look at the update log it will tell you it switched "suricata" package for "suricata-stable", so merely a switch, not really a "forced" install. Everything can be "forced" by means of dependencies, but that's exactly what it's supposed to do, maybe "enforcing" is better. ;)


Cheers,
Franco
Title: Re: 24.1 IDS breaks internet
Post by: Sensler3000 on February 15, 2024, 08:57:52 AM
Is this issue really fixed for 24.1_1? it looks like unbound still fails from time to time. I cannot reproduce it yet but i still get random dns resolution errors. All started with 24.1.
Title: Re: 24.1 IDS breaks internet
Post by: franco on February 15, 2024, 11:26:40 AM
What's the connection between Suricata and Unbound?


Cheers,
Franco
Title: Re: 24.1 IDS breaks internet
Post by: Sensler3000 on February 15, 2024, 11:32:48 AM
My bad the issue lies somewhere else, nothing to do with IDS.