Every hour: error in configd communication

Started by MTR, September 28, 2020, 03:46:42 PM

Previous topic - Next topic
Hi,

This error message comes up in my logs twice every hour, first time exactly 2 minutes past the hour, 2nd time anywhere from 15 seconds to 1m30s later:
error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out

Any ideas what causes this?

I have a similar problem, though I only see the problem surface after a scheduled job runs at 3:00.

Backend log:

2020-09-27T03:02:09 configd.py[9056] unable to sendback response [Error (1) ] for [ids][update][None] {fc8fd981-04f4-4d10-8eb3-463aec2956a0}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe
2020-09-27T03:02:09 configd.py[9056] [fc8fd981-04f4-4d10-8eb3-463aec2956a0] returned exit status 1
2020-09-27T03:00:00 configd.py[9056] [fc8fd981-04f4-4d10-8eb3-463aec2956a0] update and reload intrusion detection rules


General log:

2020-09-27T03:02:00 configctl[46393] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2020-09-27T03:00:37 /rule-updater.py[27490] version response for https://rules.emergingthreats.net/open/suricata-4.0/version.txt : 9575
2020-09-27T03:00:33 /rule-updater.py[27490] download completed for https://rulesets.opnsense.org/suricata/opnsense.uncategorized.rules
2020-09-27T03:00:31 /rule-updater.py[27490] download completed for https://rulesets.opnsense.org/suricata/opnsense.social_media.rules
2020-09-27T03:00:29 /rule-updater.py[27490] download completed for https://rulesets.opnsense.org/suricata/opnsense.messaging.rules
2020-09-27T03:00:27 /rule-updater.py[27490] download completed for https://rulesets.opnsense.org/suricata/opnsense.media_streaming.rules
2020-09-27T03:00:23 /rule-updater.py[27490] download completed for https://rulesets.opnsense.org/suricata/opnsense.mail.rules
2020-09-27T03:00:20 /rule-updater.py[27490] download completed for https://rulesets.opnsense.org/suricata/opnsense.file_transfer.rules
2020-09-27T03:00:19 /rule-updater.py[27490] download completed for https://rulesets.opnsense.org/suricata/opnsense.test.rules
2020-09-27T03:00:19 /rule-updater.py[27490] download completed for https://urlhaus.abuse.ch/downloads/ids/
2020-09-27T03:00:04 /rule-updater.py[27490] download completed for https://feodotracker.abuse.ch/downloads/feodotracker.rules
2020-09-27T03:00:03 /rule-updater.py[27490] download completed for https://sslbl.abuse.ch/blacklist/sslipblacklist.rules
2020-09-27T03:00:03 /rule-updater.py[27490] download completed for https://sslbl.abuse.ch/blacklist/sslblacklist_tls_cert.rules


I'm not sure if there is a connection :shrug:.

Looks like it's related to Sensei on my end:Sep 29 18:02:33 OPNsense.LAN configd.py[41708]: unable to sendback response [OK  ] for [sensei][periodicals][None] {f7eccd50-e251-4804-a9ce-8ca13e3afc3e}, message was Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run     self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe


After finding this in /var/log/configd/ i opened Sensei in the GUI and i got an error about Elasticsearch and Sensei being stopped because of high swap usage. I then upped the Max Swap Utilization from 30% to 50%, started Sensei again and now the error messages are gone.

Well, the hourly errors are gone after upping Sensei's Max Swap Utilization, but i've still encountered another 'error on configd communication' error. This one happend at midnight. Corresponding error in configd logs is:Sep 30 00:02:01 OPNsense.LAN configd.py[41708]: unable to sendback response [Updating OPNsense repository catalogue... Fetching meta.txz: . done Fetching packagesite.txz: .......... done Processing entries: .......... done OPNsense repository update completed. 721 packages processed. Updating SunnyValley repository catalogue... Fetching meta.txz: . done Fetching packagesite.txz: .. done Processing entries: ... done SunnyValley repository update completed. 30 packages processed. All repositories are up to date.  ] for [sensei][check-updates][['cron']] {16ae0fd9-b172-4f93-a43e-8f92502d0525}, message was Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run     self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe
Sep 30 00:02:02 OPNsense.LAN configd.py[41708]: unable to sendback response [OK  ] for [sensei][periodicals][None] {e1801052-4702-4c17-b708-3e9d528992db}, message was Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run     self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe


I'm feeling this thread is in the wrong sub forum now. Maybe a mod should move this into the Sensei sub?

Quote from: MTR on September 28, 2020, 03:46:42 PM
Hi,

This error message comes up in my logs twice every hour, first time exactly 2 minutes past the hour, 2nd time anywhere from 15 seconds to 1m30s later:
error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out

Any ideas what causes this?

i got the same error
Supermicro A2SDi-4C-HLN4F
Team Rebellion Member (sidebar / themes: tukan, cicada & vicuna)

He guys,
any news here, to get this fixed?
I still have this error and I'm running on 21.1 now and I have turned off Sensei/ElasticSearch so far. So I'm not sure, if this is the root cause or triggering it? Just asking for fresh ideas.
Running OPNsense on 4 core Intel Xeon E5506, 20GB RAM, 2x Broadcom NetXtreme II BCM5709, 4x Intel 82580
Ubench Single CPU: 307897 (0.39s)