Scheduled Jobs Multiple instances

[alasdaircr]

Hi,

I'm noticing scheduled jobs which run for longer than 1 minute are being re-triggered to start, causing duplicate processes running.

E.g. job runs every 4 hours, it takes 5 minutes to complete. Espo starts the job, then a minute later starts another instance, etc etc

What is the best way to avoid this?

[alasdaircr]

Now I'm seeing duplicate emails in Espo :-(

Looks like the checkEmailAccounts job has run multiple times too.

This is running Espo 4.7.2

Has anyone else experienced the same?

[yuri]

By design this should not be happening. Could you debug it?

[alasdaircr]

Yep. Will need to do this later today after other work. Am disabling some jobs for now.

[alasdaircr]

Can you confirm what the logic should be on a very general sense ?

If a job is marked as running what happens when the scheduler comes round again?

[yuri]

Running scheduled jobs are skipped until they are Success or Failed (controlled by jobPeriod config parameter).

[alasdaircr]

Haven't looked into actual code yet as dealing with aftermath - but our cron job wasn't triggering for a day due to me disabling it. When I've enabled the cron job, these duplicate jobs appeared.

[alasdaircr]

Hi, I've been able to reproduce this again.

Just to prove this is not crazy these messags are from the same job.

[2017-06-16 12:53:17] Espo.INFO: EVENTBRITE: Skipping attendee with no email address [] [] - this is half way through the job (which lasts a few minutes)
[2017-06-16 12:53:26] Espo.NOTICE: EVENTBRITE DAILY: Starting.. [] [] - here the same job is starting again, despite the existing job still running

The problem seems to be that if a scheduled job is running that has an execute time in the future (which happens when there are no previous logs of the Job), the second cron iteration (after 1 minute) attempts to create a new job, fails, and instead sets the existing job's status as 'Failed'. The existing process is not terminated, it's still running. I'm presuming there is a bug in the calculation of length of job before setting as failed.. but I don't know this code very well.

The next iteration of cron, (after 1 minute), the manager sees a failed job (even though it's still running), and starts up a new instance.

Then you have duplicate jobs running at the same time.

[alasdaircr]

My solution to this is to insert a dummy log record which stops the job running ahead of time. I'll see what happens then.

[alasdaircr]

In fact I'm able to reproduce this easily.

Set a cron job schedule to run daily at a time in the past.
Watch the cron start. Then next minute later, it's set as failed (because it's running and executed in the past), another job is started. Hence duplicate jobs.

[yuri]

> Set a cron job schedule to run daily at a time in the past.

Can you explain what it means please?

[alasdaircr]

I mean for instance if the time now is 12pm. Set a schedule to be 0 4 * * * - 4am

[alasdaircr]

Then a minute later that job starts running, the 'execute' time for it in job table shows the 4am time

[yuri]

I believe it was designed that execution time will be always present or future. And jobs with current execution time never be skipped unless server is down.

[yuri]

I think the fix should be that execution time be set to current time rather than 4am.

This logic isn't designed by me. I'm not sure how it should work.