Problem with duplicate message_id in email

[LuckyLouie]

Hello,
On average, once a month, we have a case that customers call us that they sent us an order by e-mail and we do not see this email in espo.
Interestingly, these email are normally visible in other mail programs, e.g. thunderbird, roundcube.

I took a closer look at the case and noticed that these messages have the same message_id as another previous email from the same client.

I have noticed that Microsoft Outlook is responsible for creating a email with an inevitable messsage_id:

https://answers.microsoft.com/en-us/msoffice/forum/all/outlook-2016-not-delivering-message-with-duplicate/2738c521-e7d1-4314-8626-2c1262363bbb

https://serverfault.com/questions/390199/two-emails-receiving-the-same-message-id

I know that Microsoft Outlook is to blame because the message_id of an email should be unique, but it's hard to explain to customers. He often gets a reply that: "Others receive e-mails from us and have never heard of such a problem"

Is there any option for such email to be displayed in espo, as is the case in e.g. thunderbird, roundcube, gmail app?

Thank you in advance,

[Enju]

I have the same problem. When clients use Outlook 2003 for sending, sometimes the message_id is duplicated and the message does not show up in espo.
yuri could you help? Now i have to use espo client + thunderbird to not miss that email :/

[LuckyLouie]

e-mail messages sent using a template in Outlook 2010 and 2016 also do not appear in the database if the client has already sent the message using the template.
Another description of the problem:

https://serverfault.com/questions/390199/two-emails-receiving-the-same-message-id

[yuri]

Please suggest a solution. How to solve the problem w/o breaking the current functionality? Espo differs from email clients as it stores emails from multiple mailboxes in a single place.

Checking additionally for other fields, like date-sent? How reliable it would be?

[LuckyLouie]

Yes. Maybe a condition: if there is more than one message in the database with the same message_id -> display the message about latest data_sent?

[yuri]

This not viable. We don't display or not display. We import or not import. If an email is being imported and there's already an email with the same message-ID, we could allow the duplicate to import if the date-send field differs.

[LuckyLouie]

If an email is being imported and there's already an email with the same message-ID, we could allow the duplicate to import if the date-send field differs.

This solution should solve the problem. Can we expect such an implementation in the future?

[yuri]

I'm hesitant to change the existing logic as it require real-life testing. I'm not sure whether some IMAP server rewrite the Date header. If so, it would be a problem

I extracted duplicate checking to the separate class https://github.com/espocrm/espocrm/b...cateFinder.php. You can customize it to check dateStart.