I've scratched my head with this problem so need your help (I`m quite a noob with freebsd but learning).
OPNsense 24.1.7_4-amd64
FreeBSD 13.2-RELEASE-p11
Problem: none of my custom jobs (system->settings->cron) are run no matter what I tried.
Description:
The cron daemon auto-starts and runs any OPNsense jobs with no problem (see it in backend logs):
# service cron status
cron is running as pid 7636.
# ps ax | grep
7636 - Ss 0:00.00 /usr/sbin/cron -s
I've added a new custom job on the system->settings->cron page to run every midnight:
0 0 * * * AcmeClient Cronjob for Certificate AutoRenewal Renew ACME
of course, I clicked the Apply button so the cron changes are reflected/logged in system->log files->backed:
2024-05-25T12:50:27 Debug configd.py OPNsense/Cron generated //var/cron/tabs/nobody
2024-05-25T12:50:27 Notice configd.py generate template container OPNsense/Cron
2024-05-25T12:50:27 Notice configd.py [1541edde-25f0-4024-9ea3-bb6e7bc58af8] generate template OPNsense/Cron
and this job is successfully added to /var/cron/tabs/nobody:
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
#
# User-defined crontab files can be loaded via /etc/cron.d
# or /usr/local/etc/cron.d and follow the same format as
# /etc/crontab, see the crontab(5) manual page.
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
#minute hour mday month wday command
# Origin/Description: AcmeClient/AcmeClient Cronjob for Certificate AutoRenewal
0 0 * * * /usr/local/sbin/configctl -d 'acmeclient cron-auto-renew'
My other discoveries:
* the /var/log/cron folder doesn't exist while 'google' says it should and contains the cron logs. Don't know whether is true but still.
* crontab -l outputs only jobs from /var/cron/tabs/root but none from /nobody:
# crontab -l
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
REQUESTS_CA_BUNDLE=/etc/ssl/cert.pem
#minute hour mday month wday command
1 * * * * (/usr/local/sbin/configctl -d syslog archive) > /dev/null
2 * * * * (/usr/local/sbin/expiretable -v -t 3600 sshlockout) > /dev/null
3 * * * * (/usr/local/sbin/expiretable -v -t 3600 virusprot) > /dev/null
4 * * * * (/usr/local/etc/rc.expireaccounts) > /dev/null
*/4 * * * * (/usr/local/sbin/ping_hosts.sh) > /dev/null
0 22 * * * (/usr/local/sbin/configctl -d firmware changelog cron) > /dev/null
0 */24 * * * (/usr/local/etc/rc.syshook.d/backup/20-rrd) > /dev/null
1 3 1 * * (/usr/local/sbin/configctl -d filter schedule bogons) > /dev/null
* * * * * (/usr/local/bin/flock -n -E 0 -o /tmp/filter_update_tables.lock /usr/local/opnsense/scripts/filter/update_tables.py) > /dev/null
Problem:
No matter what schedule I set for this job, or added other jobs that do other stuff, even if they run every minute (*/1 * * * *) - none of these custom jobs are run.
But the job is successfully run manually via configctl <command> <subcommand>!
And seems these custom jobs didn't work from the beginning (my opnsesne was installed couple months ago) because one of my jobs was checking firmware updates and I never saw the 'new version' notification shown to me when I logged in into opnsese web UI.
I.e. I've tried to add other custom jobs at ->Cron page, change their schedule to run on specific time or recurrent - they are not run nor found in the Backend logs by any keyword.
Then I added the 'shutdown host' job that runs a custom actions script to shutdown the host and .... it didn't run.
I also tried to restart the cron service, restart the opnsense host, read opnsense github issues, reddit channel, this forum, google, etc. - no luck.
It's f...ng magic that just doesn't work.
crontab -l will only show root's cron jobs, that's expected. Check if the cron job shows under crontab -l -u nobody
(I think that just shows the contents of the file you already checked so I expect it will)
Also try running it from the prompt using su -m nobody -c 'configctl acmeclient cron-auto-renew'
in case it is a permissions problem or something running the command as nobody. Not sure why it would fail as nobody but perhaps worth checking.
I have this same command as a custom cron job and it runs fine as expected.
Thanks for the quick response.
The 'crontab -l -u nobody' shows the ../cron/nobody content, thanks.
When I tried 'su -m nobody -c 'configctl acmeclient cron-auto-renew'' it gave me '/usr/local/sbin/configctl: Permission denied' although both /usr/local/sbin/configctl and ../opnsense/service/configd_ctl.py that it reference are readable for public:
lrwxr-xr-x 1 root wheel 34 May 21 10:50 /usr/local/sbin/configctl -> ../opnsense/service/configd_ctl.py
-rwxr-xr-x 1 root wheel 6081 May 21 10:49 /usr/local/opnsense/service/configd_ctl.py
I tried to grant nobody access to that file:
setfacl -m u:nobody:rx /usr/local/sbin/configctl
but got: acl_get_file() failed: Operation not supported
Did you check permissions on all directories in the path?
ls -ld / /usr /usr/local /usr/local/sbin /usr/local/opnsense /usr/local/opnsense/service
Maybe one of those has become restricted somehow. If it's to do with setfacl then I probably can't help as I'm not familiar with that.
Yes, all intermediate folders are readable by all:
# ls -ld / /usr /usr/local /usr/local/sbin /usr/local/opnsense /usr/local/opnsense/service /usr/local/sbin/configctl /usr/local/opnsense/service/configd_ctl.py
drwxr-xr-x 21 root wheel 1024 May 24 21:53 /
drwxr-xr-x 13 root wheel 512 Apr 2 15:52 /usr
drwxr-xr-x 17 root wheel 512 Apr 2 15:55 /usr/local
drwxr-xr-x 11 root wheel 512 Apr 2 15:55 /usr/local/opnsense
drwxr-xr-x 6 root wheel 512 May 21 22:27 /usr/local/opnsense/service
-rwxr-xr-x 1 root wheel 6081 May 21 10:49 /usr/local/opnsense/service/configd_ctl.py
drwxr-xr-x 2 root wheel 2560 May 21 22:27 /usr/local/sbin
lrwxr-xr-x 1 root wheel 34 May 21 10:50 /usr/local/sbin/configctl -> ../opnsense/service/configd_ctl.py
Problem FIXED. It's almost a story for the movie ))
The 'permission denied' reason was that the opnsense/service/configd_ctl.py is a python file which shebang runs /usr/local/bin/python3 but the /usr/local/bin folder lost its all/world permission (it had rwxr-x---) because of my misconfigured ansible script that has installed some stuff under the /bin folder earlier. So because of this, /usr/local/bin/python3 was unaccessible for all non-wheel users and silently failed for them when executing .py scripts.
Thank you for directing me with this mystery, @Ben S!
Glad I could help and you got it sorted. :)
Glad you got it fixed. Just out of curiosity, why do you need to schedule a cron job for the Acme client? Are you using the Acme plug-in (automatically creates the cron job)?
julsssark, I use the os-acme-client plugin to generate certificates for my domains and that job auto-renews them (they are valid only for 3 months).
The plug-in automatically creates the cron job based on the settings in Services->Acme Client->Settings->Update Schedule. I was wondering why you needed to create a custom job for it. I may have misunderstood your original post.
Yep, that job is created if you click the 'update schedule' tab in ACME settings. Apart from that, I have other manually added jobs and all of them were not working. I just started my investigation from the ACME job that was first on the list.