23.7.4 upgrade lost device connection to internet [SOLVED]

[passeri]

After 23.7.4 upgrade I can no longer access the internet from any user device.
1. Devices can ping router and anything else internal, but not on the internet.
2. If logged in to Opnsense in GUI, or via SSH, I can ping anything internal and on internet.
3. Dashboard shows correct static IP after PPOE connection.
4. Checking via a mobile (cell) device, the ISP's web page shows no outage.

My internal test opnsense after the same upgrade can still ping upstream to the main router I upgrade it first in hope of avoiding this sort of problem.

I tried shutting down crowdsec, and rebooting router.
I compared XML of prior config with that now; nothing has changed apart from disabling crowdsec rule as mentioned.

In logs I found no error message since one in late August, from bootup saying "/usr/sbin/ngtl msg igb1:setautosrc 1 returned code 71, the output was, 'ngctl: send msg: no such directory'". Given everything has operated normally meantime, this does not seem relevant.

I am not familiar with using tools like wireshark so if you advise me to do that then please provide detail or pointers.

Can someone also point me to how to revert to 23.7.3 please? I need this back up quickly.

[newsense]

Can you access https://1.1.1.1 from inside ? Can you ping google.com ?

[passeri]

Newsense: from inside, no; from SSH to router (or in OPNsense GUI) yes.

I can see a red dot at top of GUI page, next to my login name. Clicking it shows a Firewall message, copied here:

Date
Severity
Process
Line
2023-09-17T14:20:45   Error   firewall   There were error(s) loading the rules: /tmp/rules.debug:138: syntax error - The line in question reads [138]: block in log quick inet from $crowdsec_blacklists to {any} tag .CW label "e1531e9709d2a6434e7f1e113004d63e" # CrowdSec (IPv4)   
2023-09-17T14:20:45   Error   firewall   /usr/local/etc/rc.newwanip: The command '/sbin/pfctl -f /tmp/rules.debug.old' returned exit code '1', the output was '/tmp/rules.debug.old:123: syntax error /tmp/rules.debug.old:124: syntax error pfctl: Syntax error in config file: pf rules not loaded'.

[newsense]

Disable/Uninstall Crowdsec and reboot

[passeri]

I stopped CrowdSec. it started again when I rebooted. Stopped it again. Still no access.

Looking for uninstall option.... back soon.

... in Services disabled crowdsec including lapi and bouncer. Restarted, crowdsec showed as stopped, access regained

Seems crowdsec not compatible, possibly with FreeBSD update?

Thank you for your assistance, newsense. As implied by my action described in my first post, I was suspicious of CrowdSec but did not pick up on the need to disable in services rather than stop on the dashboard.

[newsense]

Code Select Expand

pkg remove os-crowdsec

or from Firmware section in GUI

[passeri]

Our posts / my edit crossed.


Good to know about the pkg remove option also, thanks.

[newsense]

Not a compatibility issue. Devices on the internal network(s) are being flagged as offenders and all traffic is dropped. Didn't have much time to look into it

[passeri]

As a reflection, I have recently installed the test router so it is not configured closely to the primary router yet, critically with respect to a Crowdsec rule. This experience shows the potential value of the test bed though -- with mirrored configuration and basic testing I would have picked this up in a safe place for analysis before rolling to the main.

It is odd that Crowdsec broke with no configuration changes, though the test bed lacked that rule to block outgoing access to anything in its blocklist, and the TB kept working from internal to upstream, as far as the main.

[mmetc]

Hi!

Did you recently add the tag ".CW" by chance? It seems like dots are not allowed in rule tags. This creates a syntax error in the rule file, and they are all loaded together by pfctl design so nothing works.

I'll validate the form field in the next version. Does it work for you if you change or remove the tag?

[passeri]

Hi mmetc

Firstly, OPNsense and CrowdSec seem both to be operating happily now.

The rule I had added (and now abandoned pending testing) blocked internal clients from accessing external sites in the crowdsec_blocklist. I did not have .CW in the rule myself.

I have logging of CrowdSec-initiated block events switched on, and events tagged with .CW in the log. That has been there all along and seems to cause no problem, nor would I expect it to given logs are written and forgotten.

The fix was I completely removed the CrowdSec package then reinstalled it from the repository.

[mmetc]

Hi mmetc

Firstly, OPNsense and CrowdSec seem both to be operating happily now.

The rule I had added (and now abandoned pending testing) blocked internal clients from accessing external sites in the crowdsec_blocklist. I did not have .CW in the rule myself.

I have logging of CrowdSec-initiated block events switched on, and events tagged with .CW in the log. That has been there all along and seems to cause no problem, nor would I expect it to given logs are written and forgotten.

The fix was I completely removed the CrowdSec package then reinstalled it from the repository.

Thanks for the update, that's definitely curious. I could trigger a syntax error by creating a ".CW" tag but if it gave no issue and you don't know where it came from...
I'll do some more testing, thanks again

[passeri]

The coincident change event was upgrade from 23.7.3 to 23.7.4, no other. It ran perfectly before and, after package reinstall, after.

[madj42]

Don't quote me on this but I swear I had the same issue with one of my internal hosts.  I was able to manually clear the block from the CrowdSec console and then identify a solution to the issue for the future.  I used the link below to create a whitelist for my internal LAN ranges.

https://hub.crowdsec.net/author/crowdsecurity/configurations/whitelists

In hindsight it might not have been the best solution as I'd probably want to subject my internal IoT ranges to filtering but still, it's at least might be some sort of starting point for you.

[Shild73]

Hi. Faced with a similar situation. The Internet is there but does not give access to .com, .net, gmail, google. CrowdSec is not installed. I checked all the settings, but I don't understand what the problem might be. Opnsense 23.7.4