Tweet
Dealing with a significant issue that is causing some major headache. We have a member type entity in EspoCRM that holds a variety of details about people. When any updates are made to a member, a webhook fires to update a separate system. This has been working fine for the 3000+ members we have.
Starting a few days ago, the webhooks repeatedly are firing for a specific member (not all members), and it is happening 20-60 times per minute, changing 2 values back and forth repeatedly. I discovered this with over 160,000 items sitting in the webhook event queue, many of them successful, but just non-stop happening.
Any idea what is going on? This really feels like a bug of some sort! I turn on the web hooks and this flood happens, I shut them off and it stops. But no other webhooks work now!
Thank you!
Starting a few days ago, the webhooks repeatedly are firing for a specific member (not all members), and it is happening 20-60 times per minute, changing 2 values back and forth repeatedly. I discovered this with over 160,000 items sitting in the webhook event queue, many of them successful, but just non-stop happening.
- I disabled the Webhook, and they stop, and I was able to mass delete the webhook event queue. As soon as I re-activate the webhook, they fire nonstop again.
- There are other components in our environment that use the EspoCRM API to communicate, so I have since disabled these thinking they were the culprit, but they were not, the issue remains.
- Looking at the 'Stream' of the member, there are over 300,000 updates there, from this repeatedly flip/flopping settings back and forth.
- I am running 8.4.2, EspoCRM in a docker. I have tried rebooting the docker, and the situation continues after reboot.
- I have checked EspoCRM error logs, but nothing there that seems relevant at all. Web logs are clean.
- This has happened once before to another user about a month ago, and disabling the webhook, clearing the queue, and re-enabling the webhook resolved the issue.
- CPU on the CRM server used to hover around 15% daily and is now 3x as high at least.
- Have also discovered new updates or create operations to other members are not firing new webhook events!
Any idea what is going on? This really feels like a bug of some sort! I turn on the web hooks and this flood happens, I shut them off and it stops. But no other webhooks work now!
Thank you!