Cron doesn't run any scheduled job

[Rumor]

I've scratched my head with this problem so need your help (I`m quite a noob with freebsd but learning).
OPNsense 24.1.7_4-amd64
FreeBSD 13.2-RELEASE-p11

Problem: none of my custom jobs (system->settings->cron) are run no matter what I tried.

Description:
The cron daemon auto-starts and runs any OPNsense jobs with no problem (see it in backend logs):

Code: [Select]

# service cron status
cron is running as pid 7636.

# ps ax | grep
 7636  -  Ss      0:00.00 /usr/sbin/cron -s

I've added a new custom job on the system->settings->cron page to run every midnight:
   0   0   *   *   *   AcmeClient Cronjob for Certificate AutoRenewal   Renew ACME

of course, I clicked the Apply button so the cron changes are reflected/logged in system->log files->backed:

Code: [Select]

2024-05-25T12:50:27 Debug configd.py OPNsense/Cron generated //var/cron/tabs/nobody
2024-05-25T12:50:27 Notice configd.py generate template container OPNsense/Cron
2024-05-25T12:50:27 Notice configd.py [1541edde-25f0-4024-9ea3-bb6e7bc58af8] generate template OPNsense/Cron

and this job is successfully added to /var/cron/tabs/nobody:

Code: [Select]

# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
#
# User-defined crontab files can be loaded via /etc/cron.d
# or /usr/local/etc/cron.d and follow the same format as
# /etc/crontab, see the crontab(5) manual page.
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
#minute hour    mday    month   wday    command
# Origin/Description: AcmeClient/AcmeClient Cronjob for Certificate AutoRenewal
0   0   *   *   *   /usr/local/sbin/configctl -d 'acmeclient cron-auto-renew'

My other discoveries:
* the /var/log/cron folder doesn't exist while 'google' says it should and contains the cron logs. Don't know whether is true but still.
* crontab -l outputs only jobs from /var/cron/tabs/root but none from /nobody:

Code: [Select]

# crontab -l
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
REQUESTS_CA_BUNDLE=/etc/ssl/cert.pem
#minute hour    mday    month   wday    command
1       *       *       *       *       (/usr/local/sbin/configctl -d syslog archive) > /dev/null
2       *       *       *       *       (/usr/local/sbin/expiretable -v -t 3600 sshlockout) > /dev/null
3       *       *       *       *       (/usr/local/sbin/expiretable -v -t 3600 virusprot) > /dev/null
4       *       *       *       *       (/usr/local/etc/rc.expireaccounts) > /dev/null
*/4     *       *       *       *       (/usr/local/sbin/ping_hosts.sh) > /dev/null
0       22      *       *       *       (/usr/local/sbin/configctl -d firmware changelog cron) > /dev/null
0       */24    *       *       *       (/usr/local/etc/rc.syshook.d/backup/20-rrd) > /dev/null
1       3       1       *       *       (/usr/local/sbin/configctl -d filter schedule bogons) > /dev/null
*       *       *       *       *       (/usr/local/bin/flock -n -E 0 -o /tmp/filter_update_tables.lock /usr/local/opnsense/scripts/filter/update_tables.py) > /dev/null

Problem:
No matter what schedule I set for this job, or added other jobs that do other stuff, even if they run every minute (*/1 * * * *) - none of these custom jobs are run.
But the job is successfully run manually via configctl <command> <subcommand>!
And seems these custom jobs didn't work from the beginning (my opnsesne was installed couple months ago) because one of my jobs was checking firmware updates and I never saw the 'new version' notification shown to me when I logged in into opnsese web UI.

I.e. I've tried to add other custom jobs at ->Cron page, change their schedule to run on specific time or recurrent - they are not run nor found in the Backend logs by any keyword.
Then I added the 'shutdown host' job that runs a custom actions script to shutdown the host and .... it didn't run.
I also tried to restart the cron service, restart the opnsense host, read opnsense github issues, reddit channel, this forum, google, etc. - no luck.

It's f...ng magic that just doesn't work.

[Ben S]

crontab -l will only show root's cron jobs, that's expected.  Check if the cron job shows under

Code: [Select]

crontab -l -u nobody
(I think that just shows the contents of the file you already checked so I expect it will)

Also try running it from the prompt using

Code: [Select]

su -m nobody -c 'configctl acmeclient cron-auto-renew'

in case it is a permissions problem or something running the command as nobody.  Not sure why it would fail as nobody but perhaps worth checking.

I have this same command as a custom cron job and it runs fine as expected.

[Rumor]

Thanks for the quick response.

The 'crontab -l -u nobody' shows the ../cron/nobody content, thanks.

When I tried 'su -m nobody -c 'configctl acmeclient cron-auto-renew'' it gave me '/usr/local/sbin/configctl: Permission denied' although both /usr/local/sbin/configctl and ../opnsense/service/configd_ctl.py that it reference are readable for public:

Code: [Select]

lrwxr-xr-x  1 root  wheel  34 May 21 10:50 /usr/local/sbin/configctl -> ../opnsense/service/configd_ctl.py
-rwxr-xr-x  1 root  wheel  6081 May 21 10:49 /usr/local/opnsense/service/configd_ctl.py

I tried to grant nobody access to that file:

Code: [Select]

setfacl -m u:nobody:rx /usr/local/sbin/configctl
but got: acl_get_file() failed: Operation not supported

[Ben S]

Did you check permissions on all directories in the path?

Code: [Select]

ls -ld / /usr /usr/local /usr/local/sbin /usr/local/opnsense /usr/local/opnsense/service
Maybe one of those has become restricted somehow.  If it's to do with setfacl then I probably can't help as I'm not familiar with that.

[Rumor]

Yes, all intermediate folders are readable by all:

Code: [Select]

# ls -ld / /usr /usr/local /usr/local/sbin /usr/local/opnsense /usr/local/opnsense/service /usr/local/sbin/configctl /usr/local/opnsense/service/configd_ctl.py
drwxr-xr-x  21 root  wheel  1024 May 24 21:53 /
drwxr-xr-x  13 root  wheel   512 Apr  2 15:52 /usr
drwxr-xr-x  17 root  wheel   512 Apr  2 15:55 /usr/local
drwxr-xr-x  11 root  wheel   512 Apr  2 15:55 /usr/local/opnsense
drwxr-xr-x   6 root  wheel   512 May 21 22:27 /usr/local/opnsense/service
-rwxr-xr-x   1 root  wheel  6081 May 21 10:49 /usr/local/opnsense/service/configd_ctl.py
drwxr-xr-x   2 root  wheel  2560 May 21 22:27 /usr/local/sbin
lrwxr-xr-x   1 root  wheel    34 May 21 10:50 /usr/local/sbin/configctl -> ../opnsense/service/configd_ctl.py

[Rumor]

Problem FIXED. It's almost a story for the movie ))
The 'permission denied' reason was that the opnsense/service/configd_ctl.py is a python file which shebang runs /usr/local/bin/python3 but the /usr/local/bin folder lost its all/world permission (it had rwxr-x---) because of my misconfigured ansible script that has installed some stuff under the /bin folder earlier. So because of this, /usr/local/bin/python3 was unaccessible for all non-wheel users and silently failed for them when executing .py scripts.

Thank you for directing me with this mystery, @Ben S!

[Ben S]

Glad I could help and you got it sorted. 

[julsssark]

Glad you got it fixed. Just out of curiosity, why do you need to schedule a cron job for the Acme client? Are you using the Acme plug-in (automatically creates the cron job)?

[Rumor]

julsssark, I use the os-acme-client plugin to generate certificates for my domains and that job auto-renews them (they are valid only for 3 months).

[julsssark]

The plug-in automatically creates the cron job based on the settings in Services->Acme Client->Settings->Update Schedule. I was wondering why you needed to create a custom job for it. I may have misunderstood your original post.

[Rumor]

Yep, that job is created if you click the 'update schedule' tab in ACME settings. Apart from that, I have other manually added jobs and all of them were not working. I just started my investigation from the ACME job that was first on the list.