Tweet
Hi EspoCRM Community,
lazovic
yuri
I'm working with a large dataset of 300,000–500,000 records in a custom entity. I have a workflow that sends an HTTP request to an external third-party API for each record to fetch and save data back into EspoCRM.
The Problem:
We tried two approaches:
Approach 1 — After Save via Mass Update: When we run a mass update on all records, the workflow fires on every record simultaneously — sending hundreds of thousands of HTTP requests in seconds. This causes the external API to rate-limit or block our server IP completely, resulting in 0 response errors and no data being saved.
Approach 2 — Scheduled Workflow with List Report: We created a scheduled workflow using a List Report as the target. However, the report itself returns 300,000+ records and the scheduled workflow tries to process all of them at once — causing the same rate limiting problem on the external API.
Both approaches result in the external API blocking our server after processing only a small portion of records.
What We Need:
We need the scheduled workflow to process a limited number of records per run — for example, 50 records per minute — instead of all records at once. This would keep us within the external API's rate limit and allow all records to be processed safely over time.
What We've Already Tried:
Our Question:
Is there any built-in way in EspoCRM or Advanced Pack to:
Any guidance from the community or EspoCRM developers would be greatly appreciated.
lazovic
yuri
I'm working with a large dataset of 300,000–500,000 records in a custom entity. I have a workflow that sends an HTTP request to an external third-party API for each record to fetch and save data back into EspoCRM.
The Problem:
We tried two approaches:
Approach 1 — After Save via Mass Update: When we run a mass update on all records, the workflow fires on every record simultaneously — sending hundreds of thousands of HTTP requests in seconds. This causes the external API to rate-limit or block our server IP completely, resulting in 0 response errors and no data being saved.
Approach 2 — Scheduled Workflow with List Report: We created a scheduled workflow using a List Report as the target. However, the report itself returns 300,000+ records and the scheduled workflow tries to process all of them at once — causing the same rate limiting problem on the external API.
Both approaches result in the external API blocking our server after processing only a small portion of records.
What We Need:
We need the scheduled workflow to process a limited number of records per run — for example, 50 records per minute — instead of all records at once. This would keep us within the external API's rate limit and allow all records to be processed safely over time.
What We've Already Tried:
- Looked for a Limit option in List Reports — not available in EspoCRM
- Checked config parameters for any rate limiting on workflows — nothing relevant
Our Question:
Is there any built-in way in EspoCRM or Advanced Pack to:
- Limit how many records a scheduled workflow processes per run?
- Process records in batches automatically — e.g. 50 records per minute until all 300,000+ are done?
- Or any official workaround using Reports, Formula, BPM, or configuration?
Any guidance from the community or EspoCRM developers would be greatly appreciated.