Advanced Pack - New Mailchimp API

[alasdaircr]

They got back to me and said they've confirmed it and will fix it. They suggest the fix would be to request the reports with a count paramater which is identical to the number of 'sent' records from the campaign detail information.

So, e.g. GET request this in a batch request: reports/<CAMPAIGNID>/email-activity?fields=emails.email_address,emails.activi ty&count=<NUMBEROFSENDS>

This way you get the whole response in one TGZ file from one batch request with no duplicates.

Please look into implementing this improvement. I don't know where you got the idea to split in 100's that is not a limit these days. And using batch requests means there's no issues with timeouts.

[alasdaircr]

I just checked the result of a reports/campaign-id/email-activity request - there IS one duplicate coming through. This seems to be an off by one fault, as it is record number 923 from the second file of total 1923 results. The duplicate is at position 0 AND 1923. I've the API support team know about this. Should be easy to fix.

However despite that, it would be relatively easy to deal with this duplicate. Fetching the results this way compared to the way you are doing it is much much easier/faster.

[alasdaircr]

In a batch request if you don't specify count, it retrieves all of the results, so you don't have to page at all.

I.e. just this

post to /batches

{ operations: [
{ 'method' => 'GET', 'path': '/reports/<MC_CAMPAIGN_ID>/sent-to'}
]}

Then monitor the batch operation, just one per campaign. And once it's finished download the result. The TGZ contains ALL the sent-to records for that campaign.

This is not how you're doing it, you're requesting batches at 100 records at a time, which makes it much much slower.

[tanya]

on the different pages email addresses can duplicate. After all pages were watched not all emails were met. Sometimes even returns internal error

[alasdaircr]

What's the error? This returned the report fine.

[tanya]

This MailChimp API has an error with sent-to reports. We had some talk with MailChimp Support and they recommended to use the batch way of synchronization, until they fix this bug.

[alasdaircr]

Looking into the code. You're fetching the sent-to report in a strange way. You grab all recipients then for each one request the sent status in batches.

Instead you could add 1 batch request which is { operations: [ {'method' => 'GET', 'path': '/reports/campaignId/sent-to'} ] }

Then poll that 1 batch (it automatically paginates for you) for the tgz of the sent-to report.

It took just 20 seconds for my batch to run, returning 1800 sent-to reports.

[tanya]

now we are working on this. Close to the release

[alasdaircr]

tanya how are you getting on with the fixes?

I've also noticed the issue with hundreds of batch jobs being created, and then MailChimp erroring that > 500 were waiting. This is for only 10 campaigns being synced with <1200 recipients to each

[lucakuehne]

this integration can not be without BatchJobs

I don't think you understand..
If I insert the code given by you, there don't apear any MailchimpBatch Jobs at all under "<URL>/#Admin/jobs"
And without the code given by you, there are created like 2 Jobs "MailchimpBatch" per Second! Is it really neccessary to create so many jobs for the sync between Mailchimp and Espo?

[tanya]

this integration can not be without BatchJobs

[lucakuehne]

As I said before: if I insert this code there aren't generated any BatchJobs at all... And the existing ones run forever and never finish

[tanya]

code, I sent you before, avoids new checking of these batches

[lucakuehne]

And there are many more of these "
MailChimp: Error after requesting GET"-Error logs until 14:10

[lucakuehne]

EspoCRM-Log (Part5):

[2017-01-18 14:00:06] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6d2d8e7453863) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/fde72a046d. The requested resource could not be found. [] []
[2017-01-18 14:00:06] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6d31a9058e6f9) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/2667445ed3. The requested resource could not be found. [] []
[2017-01-18 14:00:06] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6d31cd9b01881) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/33707773fa. The requested resource could not be found. [] []
[2017-01-18 14:00:07] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e09a9df1d25306) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/dd84b6af0e. The requested resource could not be found. [] []
[2017-01-18 14:01:02] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6d36495e4cc87) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/daf793edf0. The requested resource could not be found. [] []
[2017-01-18 14:01:02] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6d364be946120) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/88f187e31f. The requested resource could not be found. [] []
[2017-01-18 14:01:02] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6d3b63ca9784d) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/f478c152ca. The requested resource could not be found. [] []
[2017-01-18 14:01:02] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6d3bede1a26dc) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/17c85b8f2f. The requested resource could not be found. [] []
[2017-01-18 14:01:02] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6d403aabfab4e) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/b5540e000e. The requested resource could not be found. [] []
[2017-01-18 14:01:03] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6d40aa2d43166) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/dfbbcf27b2. The requested resource could not be found. [] []
[2017-01-18 14:01:03] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6d4281cac44a5) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/957ce39ea7. The requested resource could not be found. [] []
[2017-01-18 14:01:03] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6d4335fb0aab2) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/f675c46960. The requested resource could not be found. [] []
[2017-01-18 14:01:03] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e09a59cd4476ee) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/87d720f27b. The requested resource could not be found. [] []
[2017-01-18 14:01:03] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e09a10a1201b0e) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/6c2a9c4cc2. The requested resource could not be found. [] []
[2017-01-18 14:01:04] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e099c757cbb0a7) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/68072753a5. The requested resource could not be found. [] []
[2017-01-18 14:01:04] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e0997bff82a493) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/41b4a7a4ed. The requested resource could not be found. [] []
[2017-01-18 14:01:04] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e0993777f7e45e) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/c3767c7654. The requested resource could not be found. [] []
[2017-01-18 14:01:04] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e098e2be2cbcd9) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/5e9f2d53d2. The requested resource could not be found. [] []
[2017-01-18 14:01:04] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e09895a8ff8275) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/dda3e1f2fc. The requested resource could not be found. [] []
[2017-01-18 14:02:02] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e0984ddaa84d36) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/d138ad66c4. The requested resource could not be found. [] []
[2017-01-18 14:02:03] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e098082e0e5c05) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/39ab27157c. The requested resource could not be found. [] []
[2017-01-18 14:02:03] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e097bc372431a0) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/0f369a1ce4. The requested resource could not be found. [] []
[2017-01-18 14:02:03] Espo.ERROR: MailChimp (loadBatchResult - batchId 586e097707ae4394e) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/b547228135. The requested resource could not be found. [] []
[2017-01-18 14:02:03] Espo.ERROR: MailChimp (loadBatchResult - batchId 586d5776616ec24c4) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/9c9bba462d. The requested resource could not be found. [] []
[2017-01-18 14:02:03] Espo.ERROR: MailChimp (loadBatchResult - batchId 586d5773dff4f862a) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/397dad9e08. The requested resource could not be found. [] []
[2017-01-18 14:02:04] Espo.ERROR: MailChimp (loadBatchResult - batchId 586b6f9d5eb64db25) : MailChimp: Error after requesting GET https://us12.api.mailchimp.com/3.0/batches/721706c15b. The requested resource could not be found. [] []