Email Fetching Duplicate Messages

[cardmaverick]

The email fetcher is fetching the same messages over and over. I have several copies of some messages.

[yuri]

It's very unlikely. Can you check what message_id is stored in database for those emails. If they are different then email can't be treated as duplicates.

[cardmaverick]

message_id is NULL - a quick search of the real inbox shows this message only appears once. They are real duplicates.

[cardmaverick]

I just found other messages with message_id field not empty that are also duplicates.

[yuri]

In EspoCRM there is a check whether any email with the same message id exists. If exists then new record won't be created.

We don't have any ability to check whether emails are real duplicates if you have emails with different message ids on your IMAP.

[yuri]

There is the issue in Outlook (maybe some other clients too) that stored 'Sent' emails are stored with a different message id than original email. Maybe this is that case.

[joy11]

I've had the same thing happening. Duplicate and sometimes triplicate. I've had several this morning that are identical except for id field. I'm still running 4.8.4 with adv pack 1.28. We use gmail. This is my error log for today:

[2018-01-23 13:30:19] Espo.ERROR: API [GET]:.*, Params:Array ( ) , InputData: - [] []
[2018-01-23 13:30:19] Espo.ERROR: Display Error: , Code: 400 URL: /?entryPoint=attachment&a%3Clink%20href= [] []
[2018-01-23 13:30:19] Espo.ERROR: Uncaught Exception Slim\Exception\Stop: "" at /home/NAME/public_html/vendor/slim/slim/Slim/Slim.php line 1022 {"exception":"[object] (Slim\\Exception\\Stop(code: 0): at /home/NAME/public_html/vendor/slim/slim/Slim/Slim.php:1022)"} []
[2018-01-23 13:53:57] Espo.ERROR: API [GET]:/:controller/:id, Params:Array ( [controller] => Case [id] => 5a5e2a5ca7908b735 ) , InputData: - [] []
[2018-01-23 13:53:57] Espo.ERROR: Display Error: , Code: 404 URL: /api/v1/Case/5a5e2a5ca7908b735 [] []


The only true email error that I get sometimes is like this one:
[]
[2018-01-22 20:42:05] Espo.ERROR: EmailAccount 58ac52b87766cc88f (Import Message w/ MailMimeParser): [23000] SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '5a664c76d8bd02868-586e8f0f44e8f7036' for key 'UNIQ_12A5F6CCA832C1C9A76ED395' []

[cardmaverick]

I have one message repeated 6 times. This is a real bug.

[cardmaverick]

I also have hundreds of sent messages that are doubled up.

[yuri]

How can I explain. It's a bug in the mail client that stored those email copies with different message ids.

If you have such a situation then don't fetch "Sent" folder that contains email duplicates with different message id.

[joy11]

I'm not the op, but I've been having a similar issue. I have duplicate emails in EspoCRM with the same message ids and after checking the Gmail account there is only 1 email in the inbox. It seems that it's not just the other email client. It happens in both personal emails (where I could see possibly the "sent" folder being the issue) and also group email (where the "sent" is not monitored) which generates Cases and then generates multiple cases with the email being in the stream of both and attached in the activities of the last one. To me, it seems like the job is possibly taking too long and starts to fetch again. I personally haven't seen more than 3 duplicates which is the same number of times a job will try before failing, correct? Is there something I could try to increase the time between retry on the email fetching job?

I also have seen the issue where duplicate emails have been sent out of the system. There is a workflow to send out support case closed notifications and about 15 minutes ago I had the "same" notification go out twice. The content of the email was the same and same time etc. In this circumstance, the message_id was different, however.

Do you think it could it be a conflict issue with workflow generated emails and the email fetch process?

[yuri]

When an email sent twice, it could be the situation when email has been sent but SMTP server responded with an error. Then EspoCRM tries to re-send it (up to 3 times).

Do you have the latest EspoCRM version?

[joy11]

I'm in the process of updating to the newest version. I'm on 4.8.4 on the live server and running 5.0.3 on my test but having to fix some custom things that no longer work before upgrading live.

[yuri]

This bug was reported to Microsoft multiple times and they don't want to fix.

[yuri]

Do not treat EspoCRM as an ordinary mail client. It's a CRM with a different implementation than any email client.

[cardmaverick]

How can I explain. It's a bug in the mail client that stored those email copies with different message ids.

If you have such a situation then don't fetch "Sent" folder that contains email duplicates with different message id.

That's not what's happening. The message ids are identical. I'm getting duplicates of both sent messages as well as duplicates of incoming messages.

[yuri]

So you have emails with identical message ids in espocrm database?

[cardmaverick]

So you have emails with identical message ids in espocrm database?

Yes. This is a huge problem. I have literally hundreds of these. Email dates are wrong as well - I'm getting messages marked today that were received years ago. The whole thing seems to not work correctly.