HTTP Request Rate Limiting and Data Saving Issue During Large Dataset Processing

[sapyantuk]

Hello EspoCRM Team,

I am working with a large dataset import in EspoCRM. For each record, a Workflow HTTP Request is triggered to an external API endpoint.

The problem is that the endpoint server cannot handle too many API calls at once. When EspoCRM sends requests for all records simultaneously, the endpoint starts failing or slowing down. Because of this, I would like to limit the number of requests sent from EspoCRM, for example:

• Only 100 requests per minute, or
• A delay of 3–4 seconds between each HTTP request, or
• Processing requests in batches instead of sending them all at once.

Is there any built-in configuration in EspoCRM (Workflow, BPM, Scheduled Jobs, or Queue system) that allows:

1. Limiting the number of HTTP requests per minute
2. Adding a delay between HTTP requests
3. Processing records in controlled batches

---

Additional Issue


I am also facing another problem during this process.

The endpoint server actually returns the response successfully, but after a certain number of records EspoCRM stops saving the data. When this happens, I have to restart my Linode server, and after restarting it starts working again.

This behavior suggests that something may be getting blocked or overloaded on the server side (possibly cURL, PHP workers, or queue processing).

Could you please advise:

• What might cause EspoCRM to stop saving responses after many HTTP requests?
• Are there any server configurations or Espo settings that should be adjusted to prevent this issue?

[victor]

If your HTTP Request is executed using Workflow, then you can create multiple Scheduled Workflows: https://docs.espocrm.com/administrat...ows/#scheduled, which will be launched in identical batches (each batch is a separate List Report from which the Workflow is created).
Additionally, Scheduled Workflows are regulated by Administration > Scheduled Jobs > Run Scheduled Workflows.

[yuri]

You can utilize a BPM flowchart with an intermediate timer event with a random delay. There's no random number formula function out of the box but you can add it.