Jobs in queue not executing on time.

[AgentT]

Hello

I have a use case where I need to push some client information to a third party vendor using an API after the Opportunity has been created. I am using the beforeSave hook to create a job in the queue q0 which has the body defined in a Service.

Since the job is in the queue q0 which is scheduled to run every minute as mentioned here https://github.com/espocrm/espocrm/b...eduledJob.json

But sometimes the job does not get triggered immediately, but it executes 30 min or even an hour later. Seems like the cron does not seem to trigger the queue.

Is there any way to forcefully trigger jobs to be executed as and when its created?

[yuri]

Hi,

1. Try creating a job w/o queue. If you have the queue not empty, it can delay job execution, as queue is processed job-by-job.
2. Make sure you have 'Run in Parallel' parameter enabled in jobs settings.
3. Setup cron to run more often than once a minute. https://docs.espocrm.com/administrat...bs/#setting-up

[AgentT]

Thank you for the response!

1. If jobs are created without queue some of the jobs would just not run at all. And with the queue, as you said the jobs will be processed job-by-job.
2. Yes that setting is enabled but jobs in queue would still be processed one at a time.
3. I'll check this out!

[yuri]

I've never encountered the problem that jobs are never run (unless failed). They must get status Failed if they failed to run.

Jobs in the same queue are processed one-by-by by design. That's why I recommend not using queue for your case. In v7.0 there will be an additional job group parameter, making possible to have sub-queues within a queue.

[Akshay karapurkar]

Hi, i have come across similar issue where the job takes long time to cmplete

[yuri]

If you usually have too many jobs you need to tweak job settings to set it to run more jobs per one cycle. The functionality is well tested by time.

Though it was rewritten in v7.0.

[Akshay karapurkar]

Hi, i have got logs from job table of Q0, these are the logs when each time scheduled task are executed. Note: we have not done any changes in scheduled cron and using what espo is specified(* * * * this is the default what espo provides https://github.com/espocrm/espocrm/b...eduledJob.json). if you look at the logs carefully, sometimes the cron does not run every minute (look for column created_at after 18:51:12) the task was not triggered for almost 50 minutes, and after that suddenlt it started can you help us out with this we are unable to get the reason and it is

[AgentT]

Hey yuri we found the cause for the problem, and its not the problem with the cron. Its that we have a huge job with takes a long time to run (more than 30 min). It is not setup to run in any queue. This job is created, which runs from the Attachment service, which reads records from a spreadsheet file and updates certain fields under Lead entity.

You mentioned "tweak job settings set it to run more jobs per one cycle", we have parallel jobs enabled with the following settings:
Jobs Pool Concurrency Number: 20
Jobs Max Portion: 25
Daemon Interval: 10
Daemon Process Timeout: 36000
Daemon Max Process Number: 15

What do you mean by "cycle" here? Does it mean that the jobs are processed in a batch, and the next batch executes only after all the jobs in the previous batch has been completed?

We have also noticed that the logs under espocrm/data/logs/ stop when this big job is running.

FYI: We are using ESPO version 6.1.7.

Thanks for the help! :-)