Major Issue - Update webhook keeps firing repeatedly, 20-60 per minute

[Rubedizzle]

Dealing with a significant issue that is causing some major headache. We have a member type entity in EspoCRM that holds a variety of details about people. When any updates are made to a member, a webhook fires to update a separate system. This has been working fine for the 3000+ members we have.

Starting a few days ago, the webhooks repeatedly are firing for a specific member (not all members), and it is happening 20-60 times per minute, changing 2 values back and forth repeatedly. I discovered this with over 160,000 items sitting in the webhook event queue, many of them successful, but just non-stop happening.

• I disabled the Webhook, and they stop, and I was able to mass delete the webhook event queue. As soon as I re-activate the webhook, they fire nonstop again.
• There are other components in our environment that use the EspoCRM API to communicate, so I have since disabled these thinking they were the culprit, but they were not, the issue remains.
• Looking at the 'Stream' of the member, there are over 300,000 updates there, from this repeatedly flip/flopping settings back and forth.
• I am running 8.4.2, EspoCRM in a docker. I have tried rebooting the docker, and the situation continues after reboot.
• I have checked EspoCRM error logs, but nothing there that seems relevant at all. Web logs are clean.
• This has happened once before to another user about a month ago, and disabling the webhook, clearing the queue, and re-enabling the webhook resolved the issue.
• CPU on the CRM server used to hover around 15% daily and is now 3x as high at least.
• Have also discovered new updates or create operations to other members are not firing new webhook events!

Any idea what is going on? This really feels like a bug of some sort! I turn on the web hooks and this flood happens, I shut them off and it stops. But no other webhooks work now!

Thank you!

[lazovic]

Hi Rubedizzle,

Could you please show your webhook code (without privacy data)?

[Rubedizzle]

Hello lazovic ,

Not easy to share the remote webhook code (there isn't any special code on the EspoCRM for this), but effectively what the remote endpoint does is anytime a Member is created or updated, EspoCRM sends that payload to a remote Wordpress website to update accordingly via the webhooks feature. So if the Member's name or email, or address is changed in the CRM, it is updated remotely to reflect on the separate website.

Here's the thing - I have changed the URL of the update webhook here on EspoCRM as part of the testing here, so it'll fail to reach the remote website, and turned the Update webhook back on, and it still fires repeatedly, endlessly. It should only be firing when changes are being made to the Member from the CRM.

We don't use workflows at all in the CRM currently either, FYI, so no automation that could be causing an endless loop.

I created a new Member (test one) in EspoCRM, shut off the Update webhook over night, and it wasn't until 30+ minutes later that the Create webhook fired. Typically this would run in a few seconds.

This has been working for months without issue, aside from this same scenario coming up about a month ago. In that case though, we caught it quickly and the member only had about 17,000 repeats or so. We were able to shut off the Update webhook, purged the queue, and turned it back on, and it was the fine.

It feels to me like there is a queue somewhere that has built up with an extraordinaire amount of these Update webhooks, and as a result other updates for Members etc aren't getting completed quickly. Do we know what other queues / job schedulers are used, and perhaps one of those is overloaded and needs to be purged?

[lazovic]

Rubedizzle,

Can you please clarify who is changing the record (which user is specified in the Modified By field)? Which fields exactly are changing?

Also, have you looked at the access logs on the server? Perhaps other API requests affected this record.