How to debug cron jobs?

[florihupf]

Hi there,

I am trying to setup a cron job to run a small script with bash. Clicking “Run manually” works. All is well.

However, when I setup to run the script every hour, nothing happens.

I see from the logs that midctl tries to run it:

CRON[707817]: (root) CMD (PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" midclt call cronjob.run 2 true > /dev/null 2> /dev/null)

But nothing happens. Even if I put a plain echo hello > /some/path.txt in the script it won’t get executed.

I checked /var/log/middlewared and don’t see any errors either.

When I manually run the midctl command line from the cron (adding -U and -P credentials) all I get is a decimal number on stdout, but no script execution.

How can I debug this and / or what could be wrong?

This is on latest Fangtooth.

[Arwen]

The most common problem I have seen with Cronjob failures, is that the executable PATH is not listed. Cronjobs ignore the user’s SHELL Profile, RC file or Login file and only use the OS supplied PATH and SHELL parameters.

So this would fail:

#!/bin/bash
#
myprog param >>myprog.log

While this would work:

#!/bin/bash
#
/mnt/pool/home/myprog param >>/mnt/pool/home/myprog.log

Obviously optimizations can be made:

PATH=${PATH}:/mnt/pool/home
LOGFILE=/mnt/pool/home/`basename`.log

Next most common Cronjob failure is permissions. Either execute & read permissions of the script, or it’s directory when not run as root.

If it is really not running anything, then you may need to use set -x in the beginning of the script. Then make sure you capture all it’s output:

touch /mnt/pool/home.mycron.log
mycron 2>&1 >>/mnt/pool/home.mycron.log

[florihupf]

Hi Arwen,

Thanks for the pointers. I am still at a loss. To trim down the variables my cron job now is run by root and all it does is /usr/bin/touch /root/hello.txt.

But again, this doesn’t produce /root/hello.txt even though journalctl says:

Sep 29 07:13:01 glowy CRON[1377115]: (root) CMD (PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" midclt call cronjob.run 3 true > /dev/null 2> /dev/null)

And still, clicking “Run now” works just fine.

Is there a way to debug this further?

[bacon]

The “decimal number” you get is the job id. You can check the result in the WebUI:

Maybe you could share a screenshot of your cron job?

[florihupf]

Ok, thanks for pointer re. decimal number. I was digging more and when I run the cron job manually via midctrl things work and the job is executed.

When I just setup the time (say for debugging every nth minute after the hour) journalctl says it ran midctrl but there is no new entry in the jobs page.

So it looks like that midctl fails to issue the actual cronjob when it is ran via cron itself. Manual running (and of course adding -U and -P credentials) works.

I noticed that the midctl command shown via journalctl lacks the -U and -P options:

Here my jobs definition:

And the success message, IF run manually.

Also thanks for the set -x pointer. Very helpful.

[bacon]

If you run midclt as root you don’t need any authentication. Or at least you shoudn’t.

What happens if you run midclt as root WITHOUT -U and -P?
Try this:

sudo midclt call cronjob.run 3 true

[florihupf]

Aha! It isn’t happy: Not authenticated

[florihupf]

Ok,

Thanks @bacon you pointed me into the right direction. Looks like my root user was not part of the builtin-administrators group.

Adding root to that group fixed it (looks like, need to verify a bit more).

⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⠴⠒⠒⠒⢤⡀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⠎⠁⠀⠀⠀⠀⠀⡧⠀⠀⠀⠀⠀
⠀⡴⠚⠛⠛⠛⢲⢶⡾⠧⠔⠂⠻⣄⣀⣀⠴⠁⡴⠂⠀⠀⠀
⠘⣄⡀⠀⠀⣰⢇⣞⣁⠀⣀⣀⡀⣀⣀⣤⢀⣾⡡⡄⠀⠀⠀
⠀⠈⠉⠀⣰⡟⡼⢻⣏⣼⣵⣏⠞⡿⢋⣿⣾⠻⡏⠁⠀⢰⡆
⠀⣄⠀⣰⠟⠀⠀⠀⣀⠀⡀⢀⣀⠀⠀⠀⠁⠀⠙⢦⣀⠜⠁
⠈⠓⠒⠁⠀⠀⠀⣼⢃⡾⣷⠋⣿⣴⢋⣴⠋⠀⠀⠀⠀⠀⠀
⠀⠀⣀⠤⢤⣤⣀⣛⣽⠃⠳⠞⠁⠓⠊⢁⣠⠞⠀⠀⠀⠀⠀
⠀⠘⡅⠀⠀⠀⢈⡿⠛⠳⠦⢤⣀⣀⡀⠀⠀⣀⠀⠀⠀⠀⠀
⠀⠀⠙⠦⠤⠔⠋⠀⠀⠀⠀⠀⠀⠉⠉⠉⠉⠀⠀⠀⠀⠀⠀

[Arwen]

Ah, I should have remembered the 3rd most common Cronjob failure, the lack of cron permissions for that user.