Sensei Bug?

[IsaacFL]

Just installed Sensei today and now I notice the following error in the General Log:

Code: [Select]

2021-07-01T20:03:00 configctl[52308] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T20:03:00 configctl[57681] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out

I saw a prior post in the forum that marked it as solved but still seeing it every 10 minutes.

Sensei version 1.9.1

[annoniempjuh]

i got the same messages in the logbook, i "fixed" it with the following settings:

edit: this was not the sollution!
-Max Swap Utilization (% of total SWAP):
    'default' (30%?) to 65%
- Size of the Fast Temporary Memory Disk (in Megabytes / Capacity ):
    from 'default' (50) to 494 (512) <- why not 500 allowed? GUI says 500 is the limit but i can't set this to it...)

weirdest thing, system resources are almost nothing:
Memory usage:  31 % ( 5106/16274 MB )
SWAP usage:    0 % ( 0/8192 MB )
Disk usage    0% /usr/local/sensei/output/active/temp [ufs] (24K/496M)

but the error messages are gone after change those 2 settings, no reboot needed.

Code: [Select]

2021-07-01T11:06:00 configctl[20370] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T11:06:00 configctl[90927] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:56:00 configctl[32824] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:56:00 configctl[79480] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:46:00 configctl[85775] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:46:00 configctl[42332] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:36:00 configctl[6451] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:36:00 configctl[27432] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:26:00 configctl[13771] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:26:00 configctl[95909] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:16:00 configctl[90491] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:16:00 configctl[76087] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:06:00 configctl[68370] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T10:06:00 configctl[14388] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:56:00 configctl[46857] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:56:00 configctl[38301] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:46:00 configctl[66205] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:46:00 configctl[75635] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:36:00 configctl[69777] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:36:00 configctl[4108] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:26:00 configctl[98505] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:26:00 configctl[29457] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:16:00 configctl[40740] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:16:00 configctl[38598] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:06:00 configctl[84854] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-01T09:06:00 configctl[2654] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out


[IsaacFL]

I tried this and it didn't fix it for me.  I did notice that the increase in memory usuage did not take affect until after a reboot.

I get the error every 10 minutes, always on the 3 minute (ie Jul 02 09:23:00, Jul 02 09:33:00) so must be in a cron job.

[annoniempjuh]

I tried this and it didn't fix it for me.  I did notice that the increase in memory usuage did not take affect until after a reboot.

I get the error every 10 minutes, always on the 3 minute (ie Jul 02 09:23:00, Jul 02 09:33:00) so must be in a cron job.

it did fix it for my until i had to reboot opnsense, now Elasticsearch won't start anymore, removed sensei and started with an Elasticsearch DB somewhere else... no luck for me, the error message are back, feel pretty bad about it, especially it started AFTER i got a License key, hope the sensei team give some input, this is freaking annoying me!

edit: even a full reinstall (did use the config backup, with all Sensei data removed from it) and after setting everything back the error got back, fully remove Sensei stops the error message.
gonna leave it to rest now so i can create tomorrow a bug report with some "clean" logs.

[sy]

Hi,

It can be related to a cron job. Looking into it.

[IsaacFL]

I uninstalled Sensei and the error message has not reoccurred since.  So something to do with the Sensei install.

I am going to try to reinstall Sensei later and see if it comes back, but first want to make sure nothing else is causing the messages.

[sy]

Hi,

@IsaacFL, @annoniempjuh Please can you send us a bug report from the upper right corner of SenseiGUI.

[annoniempjuh]

Hi,

@IsaacFL, @annoniempjuh Please can you send us a bug report from the upper right corner of SenseiGUI.

just did

[IsaacFL]

Reinstalled, and still same error, just on the 5 mark now.

Code: [Select]

2021-07-02T14:55:00 configctl[31940] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-02T14:55:00 configctl[69443] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-02T14:45:00 configctl[72070] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-02T14:45:00 configctl[31411] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
Just sent Bug Report

[annoniempjuh]

after update sensei to 1.9.2 i saw an another error in the logbook:

Code: [Select]

2021-07-07T11:15:34 configd.py[71412] unable to sendback response [OK ] for [sensei][check-health][None] {5e9f5b34-d4fa-4ba8-8602-779a0a7cce88}, message was Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe

[annoniempjuh]

Updated OPNsense to 21.1.9 and the "problem" is back:

Code: [Select]

2021-07-28T11:02:21 configctl[78420] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T11:02:00 configctl[11422] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T10:02:08 configctl[94643] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T10:02:00 configctl[95994] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T09:02:08 configctl[4214] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T09:02:00 configctl[78033] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T08:02:08 configctl[28825] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T08:02:00 configctl[56539] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T07:02:07 configctl[61912] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T07:02:00 configctl[17779] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T06:02:24 configctl[53227] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T06:02:00 configctl[80721] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T05:02:07 configctl[23515] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2021-07-28T05:02:00 configctl[13985] error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
edit: looks like there got something wrong with updating OPNsense, Sensei got stopped because the SWAP was over 50% and i found some more errors in the syslogbook:

Code: [Select]

2021-07-27T18:37:02 kernel swap_pager_getswapspace(9): failed
2021-07-27T18:37:02 kernel swap_pager_getswapspace(18): failed
2021-07-27T18:37:02 kernel swap_pager_getswapspace(20): failed
2021-07-27T18:37:02 kernel swap_pager_getswapspace(3): failed
2021-07-27T18:37:02 kernel swap_pager_getswapspace(9): failed
2021-07-27T18:37:02 kernel swap_pager_getswapspace(18): failed
2021-07-27T18:37:02 kernel swap_pager_getswapspace(20): failed
2021-07-27T18:37:02 kernel swap_pager_getswapspace(3): failed
2021-07-27T18:37:02 kernel swap_pager_getswapspace(9): failed

Code: [Select]

configctl[49709] event @ 1627401591.49 msg:
2021-07-27T18:08:11 configctl[49709] event @ 1627401591.49 msg:
2021-07-27T18:08:11 configctl[49709] event @ 1627401591.49 msg:
2021-07-27T18:08:11 configctl[49709] event @ 1627401591.49 msg:
2021-07-27T18:08:11 configctl[49709] event @ 1627401591.49 msg:
2021-07-27T18:08:11 configctl[49709] event @ 1627401591.49 msg:
2021-07-27T18:08:11 configctl[49709] event @ 1627401591.49 msg:
2021-07-27T18:08:11 configctl[49709] event @ 1627401591.49 msg:
2021-07-27T18:08:11 configctl[49709] event @ 1627401591.49 msg:
2021-07-27T18:08:11 configctl[49709] event @ 1627401591.49 msg:
2021-07-27T18:08:11 configctl[49709] event @ 1627401591.49 msg:
just rebooted OPNsense and Sensei did startup, going to monitor this and wil report later this day.

[sy]

Hi,

It is fixed for 1.9.3 and it is available for 21.7 series. There is a problem with OPNsense update now. When it fixed please update both OPNsense and Sensei.

[Mks]

Hi,

encountered the same issue on most recent version:

OpnSense Version:
OPNsense 21.7.3_3-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
OpenSSL 1.1.1l 24 Aug 2021


Sensei version:
Engine Version:    1.9.3
UI Version:    21.9.14
Database Version:    1.9.21091012

I filed a bug report to Sensei.

br

[sy]

Hi,

Looking into the logs.

[eponymous]

Hi,

I'm also seeing the exact same errors as in the first post above.

However, it only happens once per day for me rather than every 10 minutes.

Seems to have been reported by numerous people:

https://forum.opnsense.org/index.php?topic=23120
https://forum.opnsense.org/index.php?topic=19348
https://forum.opnsense.org/index.php?topic=20923

I'm also getting a follow-up message (possibly unrelated) except mine is:

Code: [Select]

configd.py[82227]: unable to sendback response [OK  ] for [ids][update][None] {c02da2ca-fdd0-44c2-bb74-4861e693fa62}, message was Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/processhandler.py", line 202, in run     self.connection.sendall(('%s\n' % result).encode()) BrokenPipeError: [Errno 32] Broken pipe

I'm on the latest OPNsense and Sensei.

@sy did you make any headway on finding out the root cause?

Is there anything to be worried about here?