since the last update to 22.1.1_1 with no config change I'm getting very strange behavior of the system. Are others experiencing this also?
some apps on the iPhones on wifi don't load any more, some websites don't load, some are.
I can't not find why. And errors in the system-log-backend:
configd.py unable to sendback response [Updating OPNsense repository catalogue... Fetching meta.conf: . done Fetching packagesite.txz: .......... done Processing entries: .......... done OPNsense repository update completed. 779 packages processed. Updating SunnyValley repository catalogue... Fetching meta.conf: . done Fetching packagesite.txz: .. done Processing entries: .... done SunnyValley repository update completed. 32 packages processed. Updating mimugmail repository catalogue... Fetching meta.conf: . done Fetching packagesite.txz: ....... done Processing entries: .......... done mimugmail repository update completed. 170 packages processed. All repositories are up to date. ] for [sensei][check-updates][['cron']] {69965927-38b8-4217-8dd0-5d75994f6308}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe
Error configd.py Timeout (120) executing : firmware remote
Error configd.py [f0aef359-a298-4aa7-8b71-3fccbf91beb9] Script action stderr returned "b'[07-Feb-2022 08:58:51] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful\n\n[07-Feb-2022 08:58:52] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful\n\nnginx: the configuration file /usr/local/etc/nginx/nginx.co'"
More errors in system-log file-general:
2022-02-17T13:58:00 Error opnsense /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface WAN.
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: The RAMVPN_GW monitor address is empty, skipping.
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: The WAN_DHCP monitor address is empty, skipping.
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: The WAN_DHCP6 monitor address is empty, skipping.
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway 'fe80::xxxxx %igb0'
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv6 default route to fe80::xxxx
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: IPv6 default gateway set to wan
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway '217.xxxxxc'
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv4 default route to 217.xxxxx.1
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: IPv4 default gateway set to wan
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: On (IP address: 2001:xxx.xxxxx) (interface: WAN[wan]) (real interface: igb0).
2022-02-17T13:57:59 Error opnsense /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'igb0'
2022-02-17T13:57:58 Error opnsense /usr/local/etc/rc.linkup: The command '/usr/local/opnsense/scripts/dns/unbound_dhcpd.py --domain 'localdomain'' returned exit code '1', the output was 'Unable to lock on the pidfile.'
Is this ipv6 related? What is going on with the latest opnsense version? Help is much appreciated. Even a site like https://www.dnsleaktest.com/ won't load but other sites do... very strange...,
After a reboot of opnsense it all seem to work, and than after some minutes all te strange behavior starts again
Same, logs are ALL screwed up, info is logged as ERROR, half my clients no longer work but hard to tell what's up because...logging. Im going to back this one out probably as this was one of the worst updates in recent history. Assuming the move to bsd13 was the issue here, as always should have waited a few months for the point releases to fix everything that broke :(
I can confirm this - many errors in all system logs.
Qualified reports people, this is not helpful.
Cheers,
Franco
I would love to isolate the problem more....but could not for now.
That is why I posted a selection from the errors log.
strange thing is that after opnsense reboot it works and very fast there after, just a couple of minutes, it gets acting weird again. Any suggestion where I should start looking for clues?
Hello
I have the same feeling on my side.
Checked the interface and MTU but everything seems ok
Hardware : PCengine APU2C4
Was working as a charm on 22.1 GA
Logan
I too found this after an upgrade from 22.1 to 22.1.1-1... After the initial reboot nothing outside of my LAN would load (as if the firewall or dns wasn't working correctly)... I reloaded pf service and got partial working behavior, but 'some' things were extremely slow. An additional reboot cleared everything up.
deleted
Quote from: isamudaison on February 17, 2022, 06:37:30 PM
I too found this after an upgrade from 22.1 to 22.1.1-1... After the initial reboot nothing outside of my LAN would load (as if the firewall or dns wasn't working correctly)... I reloaded pf service and got partial working behavior, but 'some' things were extremely slow. An additional reboot cleared everything up.
The .1 update is what did it for me I think...
I had to re-install the whole OS, I would highly recommend that. Give you an opportunity to use zfs. :)
After a wipe and re-install, everything is back to normal and functioning besides sensei which was broken by the python upgrade. You have to do a wipe and re-install to fix your issues unfortunately, the .1 completely broke my network and I don't really have anything fancy. Make a config backup, reinstall, install your plugins again, reboot and restore the config. It goes very fast, the config backup is a great tool. This is why I only upgrade at my home network :D
you triggered me with sensei...
so I did a restore my opnsense config of the latest config I have and was working before updating. This way I can rule out that the update somehow did change something in the config... - same result.
than I did zenarmour reinstall with terminal:
rm -f /usr/local/sensei/etc/.configdone
and than ran the installation with the gui. Sensei/zenarmour is running, no errors.
but the strange behaviors of opnsense remains....
One one hand we might have a cosmetic problem here. I don't have any problem with my firewall malfunctioning but after reading these posts I just checked and - behold - benign events are logged with a severity of "Error". Obviously none of these events is an error.
So if I was experiencing unexplainable "weird" behaviour of my firewall, go to the log files and see nothing but "Error Error Error ..." that would make me think there is a connection, too.
Kind regards,
Patrick
@Franco
After trying by trial and error a lot of things I found in Suricata this error:
Stats for 'igb0^': pkts: 0, drop: 0 (nan%), invalid chksum: 0
I did a google, found an old opnsense forum mentioning something about setting Suricata Pattern matcher from hyperscan (what I use on my opnsense Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz (4 cores, 4 threads)
changed it to Aho-Corasick
saved and than the weird stuff ended, all working as should... but again.. after just some minute(s) all strange things are back.
changed the setting back to Hyperscan,
saved and than the weird stuff ended again, all working as should. but again, after just some minute(s) strange things as described started again.
So I think with my trial and error, it seems related to Suricata? Or changing config in Suricata refreshes "something" in opsense what "solves" the problem for some minutes....
Hope this helps the searching direction...
I have a post in the sensei section but similar, config.py is leading me to believe an issue with the way the system uses python.
Quote from: RamSense on February 17, 2022, 09:41:00 PM
@Franco
After trying by trial and error a lot of things I found in Suricata this error:
Stats for 'igb0^': pkts: 0, drop: 0 (nan%), invalid chksum: 0
I did a google, found an old opnsense forum mentioning something about setting Suricata Pattern matcher from hyperscan (what I use on my opnsense Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz (4 cores, 4 threads)
changed it to Aho-Corasick
saved and than the weird stuff ended, all working as should... but again.. after just some minute(s) all strange things are back.
changed the setting back to Hyperscan,
saved and than the weird stuff ended again, all working as should. but again, after just some minute(s) strange things as described started again.
So I think with my trial and error, it seems related to Suricata? Or changing config in Suricata refreshes "something" in opsense what "solves" the problem for some minutes....
Hope this helps the searching direction...
I'm having similar errors, but I'm not using Suricata. I am using Zen Armor though for LAN.
Posted in a different thread earlier today of strange issues with Suricata shutting off in IPS mode shortly after it starts up. Tried changing from Hybrid mode and that did not change anything either. Rebooted many times in between as well. Verified WAN IP was properly entered as well This is the log entry I am seeing:
Error suricata [116410] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - opening devname netmap:ix0/R failed: Device busy
I can reduce Suricata service to IDS only mode and I don't see this error in the log.
Running UNBOUND with DLS/TLS without issue. Also running ZENARMOR without issue.
No VLANS so no Promiscuous Mode.
to fix the current strange behavior it works with:
Suricata
Intrucion detection - administration - settings - uncheck IPS mode
Intrucion detection - administration - settings - uncheck Promiscuous mode
so the problem seems to be there indeed.
P.s. opnsense 22.1.1_3: problem still there
Can you check if it might be related to the topic raised here:
https://forum.opnsense.org/index.php?topic=26583.15
Do you have any VLANs? With this in my setting, I have major problems with scuritata, but also sensei is not running 100% stable. So currently I run with both deactivated.
seems like the same category.
I do not use VLAN's so I disabled/unchecked Promiscuous mode
after safe, all works for some time, but after a few minutes back to weirdness again. only when IPS mode unchecked it keeps stable...
Suricata works fine for me but sensei does not, suricata runs on my WAN only so no vlans which explains why it works. Sensei DOES run on vlans (LAN network, so it runs on all vlans I have under LAN). So maybe the issue is with vlan handling in the new update...hmmm
wel unfortunately I run suricarta also only on WAN and have the problems. So I can confirm it is not VLAN only.
Sensei/Zenarmor running on LAN did not automatically start, but works after manually starting. No problems there other than not auto booting.
When I see this mentioning in the system - log files - general:
Critical dhclient exiting.
Error dhclient connection closed
the strange behavior starts
---
Looking and searching all I can on google and opnsense etc.. found several mentioning errors like this in previous version and regarding Suricata / lost WAN / .. seems a bit the same, but I have no clue or knowledge how to fix this other than to shutdown Suricata....
I noticed this error now in the log:
unexpected result from https://opnsense.emergingthreats.net/api/v1/event (http_code 502)
When googling I found 1 mention with error code 403 about the Suricata token - check validity
could this be related?
---
when weird things happen and I go to terminal
ping pkg.opnsense.org -> works
pkg update -f -> works
pkg upgrade -n -> works
when fetch https://pkg.opnsense.org/FreeBSD:11:amd64/18.1/sets/changelog.txz.sig
it stalls. when I go back to the gui to Suricata and change a config setting (so that I have the system back on for a minute or so) and go to terminal fetch https://pkg.opnsense.org/FreeBSD:11:amd64/18.1/sets/changelog.txz.sig it works
1332 B 20 MBps 00s
seems the error with Suricata and brokenpipe was there in 2020?:
https://forum.opnsense.org/index.php?topic=19432.0
(https://forum.opnsense.org/index.php?topic=19432.0)
so with my knowledge I come to the conclusion that Suricata is blocking / causing the errors after a minute or so. That is also why all is solved when unchecking Suricata ips... But how to go further from here to solve this?
N.B. Did drastic fresh install from terminal with opnsense-bootstrap, system back up and running, but problem still there... I'm out of options other than the conclusion it is related to Suricata with the latest opnsense
Fixed it somewhat. Hope it helps others also! (and hopefully not necessary to do the opnsense-bootstrap step)
I did turnoff Suricata
Than I did a full download & update rules ( I noticed those were still from feb the 16th and did not get updated anymore)
after that I started Suricata with IPS enabled
System running since than!
I still noticed this error in the log:
/send_telemetry.py unexpected result from https://opnsense.emergingthreats.net/api/v1/event (http_code 502)
2022-02-19T14:04:59 Error configd.py unable to sendback response [OK ] for [ids][restart][None] {1888eb52-63ee-4e3f-a33b-cb3f954979b7}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe
2022-02-19T14:04:48 Error configd.py Timeout (120) executing : 'ids' restart
So I finally figured it out! In the recent OPNSense update 22.1.1_3, it dumped the Protected Interface under Zen Armor. Once I reset the interface to LAN, the Suricata IPS setting now sticks to On!! Now I wonder what else has not carried over from before the update....