Forum archive
Cron jobs getting executed off schedule, multiple times
I am having a problem it's taken me a week to track down. I finally figured out how to use the top command to properly identify programs that are running when my server pins itself and becomes unresponsive.
I'm using the Turnkey LAMP 12 install, running on Amazon AWS EC2.
It appears that one of my Cron jobs is being executed off schedule, and multiple (10!) instances of this same PHP process were getting kicked off at the same time. What could cause this to happen? When this occurs, my server gets pinned with 100% usage until I restart the server, or today what I did was manually kill each runaway process... which instantly returned the server to normal levels.
The reason I believe that this is the cronjob getting run off-schedule is because the only other way this PHP script runs is when I manually kick it off. It's also being executed exactly as I have it spelled out in the cron job webmin page for this job.
The times that this stuff kicks off seems totally random. Earlier this week I had it happen at aproximately 10pm, and today it happened at 4pm. The job that it's calling multiple times is scheduled to run ONCE daily at 6pm.
Replies (3)
I experienced the same issue
Reply 1Let me guess - this is a micro instance?
Reply 2A quick glance at your Webmin cron schedule it looks like it is not configured to run once at 6pm, but to run once a minute from 6pm to 6.59pm (see the left hand column: minutes - all; should be selected - 0).
And if I am correct in guessing that you are running in a Micro instance, I suspect the load is causing your CPU to be throttled (Amazon throttle micro instances pretty heavily when they exceed their CPU burst load limit).
If I'm right then that explains why you are getting multiple jobs running. And if the throttling is so heavy as to make your server pretty much grind to a halt, then that might also explain why it seems to be running at random times (the load is that high and the throttling so limiting that hours after the cron job should have finished it's still slogging away).
Jeremy, Thanks for confirming
Reply 3Jeremy,
Thanks for confirming what I figured out over the weekend. I had things configured to go off at a specific hour, but didn't specify a minute so it executed EVERY minute that hour. I recently changed from one LAMP server to an Apache server and seperate MySQL server, and when I did this I didn't set up the DB backup properly... which is why I experienced the same issue.
I had set this process up to create logfiles every time it ran... and I had 60 logfiles for the day I saw this happening... it was the smoking gun that pointed me to figure out what I'd done wrong.
I experienced the same issue with my MySQL box overnight, here is a screenshot of the TOP command... it appears instead of my backup command being run once, it was executed 28(!) times. The server becomes completely unstable and I need to manually restart MySQL to get thing working again.