cron.php hung for hours with jobRunInParallel: possibly a race in spatie/async?

[berlinerkind]

Hi everyone,

we ran into a stuck cron.php today and I'd like to share what we saw, in case it helps or someone has seen it before. I may well be missing something, so I'd appreciate any corrections.

Environment

• EspoCRM 10.0.9 (official Docker image), PHP 8.4.26 CLI
• spatie/async 1.8.1
• jobRunInParallel = true, jobPoolConcurrencyNumber = 8, jobMaxPortion = 15
• cron.php is started every 60 s by a simple loop (while true; php cron.php; sleep 60; done)

What we observed

• One cron.php started and did not return. Since our loop only starts the next run after it returns, no scheduled jobs ran for about 4 hours (email fetching, reminders, notifications).
• As far as we can tell, all 14 jobs it had started finished: the job records show Success, and no child or zombie process was left.
• The process seemed to be sleeping in the Pool::wait() loop.
• Sending it kill -TERM resolved it, and the following runs were normal.
• If we understand correctly, it would otherwise have ended only after jobPeriodForActiveProcess (36000 s), which is passed to the pool as its timeout.

A guess at the cause (not confirmed)

While reading Spatie\Async\PoolutInProgress(), I noticed the child process is started before it is added to inProgress:

$process->start();
unset($this->queue[$process->getId()]);
$this->inProgress[$process->getPid()] = $process; 

With pcntl_async_signals(true), I wonder whether the SIGCHLD handler could run in between. If a child exited very quickly, handleFinishedProcess() might not find it in inProgress yet, and the process would then stay registered as running. That would match what we saw, but I couldn't reproduce it, and it has happened only once in about 30 days, so I may be wrong.

We now have wrapped the call as timeout --foreground -k 60 2700 php cron.php.

Thanks a lot for EspoCRM and for taking a look!