Email duplicates created by system

**yuri** · 11-09-2022, 07:59 AM

Hi Jakub,

Check what message-IDs imported duplicates have (table email > column message_id). It could be that your SMTP (or maybe IMAP?) changes message IDs and Espo can't detect that it's a duplicate. Espo checks duplicates only by message-ID.

**yuri** · 11-09-2022, 08:05 AM

There will be the ability to bind a custom implementation of the duplicate finder in the next version.

https://github.com/espocrm/espocrm/b...nding.php#L249

https://github.com/espocrm/espocrm/b...cateFinder.php

**yuri** · 11-09-2022, 08:26 AM

Likely, that SMTP adds a header with an original message-ID (Microsoft adds X-Microsoft-Original-Message-ID header). The idea is to check this ID too.

PHP Code:

    public function find(Email $email, Message $message): ?Email
    {
        $messageIds = [];

        if ($email->getMessageId()) {
            $messageIds[] = $email->getMessageId();
        }

        if ($message->getHeader('X-Microsoft-Original-Message-ID')) {
            $messageIds[] = $message->getHeader('X-Microsoft-Original-Message-ID');
        }

        if ($messageIds === []) {
            return null;    
        }

        return $this->entityManager
            ->getRDBRepositoryByClass(Email::class)
            ->select(['id', 'status'])
            ->where([
                'messageId' => $messageIds,
            ])
            ->findOne();
    }

But this will be possible only in v7.3.

**Jakub Grufik** · 11-09-2022, 09:10 AM

Hello,
yeah both emails have different message_id.

I am not sure if I understand correctly, but at the moment we are not able to fix this unless the email provider stops changing the ids of the message(that is probably not going to happen), correct?

**yuri** · 11-09-2022, 09:20 AM

Correct. Without making modifications to code (that won't work after upgrade) there's no way. But as of v7.3 there's a way. Can you check a message original (raw contents of an email) of any of such emails (in any email client or web interface of your mail provider). Whether it contains an original message ID header.

**Jakub Grufik** · 11-10-2022, 09:59 AM

Hey yuri thanks a lot for the explanation and information.

Will pass this info to our email administrator to find out what could be done about this behavior.

Have a nice day sir!

**Jakub Grufik** · 04-04-2023, 07:04 AM

Hello guys, I would like to re-open this topic as our system is still being overloaded by duplicates of every single email sent within CRM.

If it goes from outlook or Gmail etc. to CRM there is no duplicate. But when we send an email from CRM to CRM (internally), a duplicate with the status "Archived" is created every single time.

Is there already any way how to solve this behavior, please?

Thanks a lot

**yuri** · 04-04-2023, 07:33 AM

Can you provide an email original (raw content) of a duplicate email?

Looks like your SMTP is changing the message ID. Maybe the original message-ID is preserved in some header.

**Jakub Grufik** · 04-04-2023, 07:51 AM

Yes, seems like smtp is changing the message id:

JSON of email with status sent:

Code:

{
   "id":"64085e55a2c6a5bc4",
   "name":"ble",
   "deleted":false,
   "subject":"ble",
   "fromName":"Klička Tomáš",
   "fromAddress":"tomas.klicka@***",
   "fromString":"Klička Tomáš <tomas.klicka@***>",
   "replyToString":null,
   "replyToName":null,
   "replyToAddress":null,
   "addressNameMap":null,
   "from":"tomas.klicka@***",
   "to":"jakub.grufik@***",
   "cc":"",
   "bcc":"",
   "replyTo":"",
   "isRead":true,
   "isReplied":false,
   "isImportant":false,
   "inTrash":false,
   "isUsers":true,
   "nameHash":{
      "tomas.klicka***":"Klička Tomáš",
      "jakub.grufik@***":"Admin Jakub"
   },
   "typeHash":{
      "tomas.klicka@***":"User",
      "jakub.grufik@***":"User"
   },
   "idHash":{
      "tomas.klicka@***":"63ff12476ca2186d7",
      "jakub.grufik@***":"630c8dc00dd195da1"
   },
   "messageId":"<4e8a1f3702ea40975a6bd7b06e558498/1678270037/1848@espo>",
   "messageIdInternal":null,
   "bodyPlain":"bla\n\n",
   "body":"<p>bla</p>",
   "isHtml":true,
   "status":"Sent",
   "hasAttachment":false,
   "dateSent":"2023-03-08 10:07:19",
   "deliveryDate":null,
   "createdAt":"2023-03-08 10:07:17",
   "modifiedAt":"2023-03-08 10:07:19",
   "isSystem":false,
   "icsContents":null,
   "icsEventUid":null,
   "fromEmailAddressId":"63ff07fa36ea149d4",
   "fromEmailAddressName":"tomas.klicka@***",
   "toEmailAddressesIds":[
      "6363bf5c22f7efaf0"
   ],
   "toEmailAddressesNames":{
      "6363bf5c22f7efaf0":"jakub.grufik@***"
   },
   "ccEmailAddressesIds":[

   ],
   "ccEmailAddressesNames":{

   },
   "replyToEmailAddressesIds":[

   ],
   "replyToEmailAddressesNames":{

   },
   "attachmentsIds":[

   ],
   "attachmentsNames":{

   },
   "parentId":null,
   "parentType":null,
   "parentName":null,
   "createdById":"63ff12476ca2186d7",
   "createdByName":"Klička Tomáš",
   "sentById":"63ff12476ca2186d7",
   "sentByName":"Klička Tomáš",
   "modifiedById":"63ff12476ca2186d7",
   "modifiedByName":"Klička Tomáš",
   "assignedUserId":null,
   "assignedUserName":null,
   "repliedId":null,
   "repliedName":null,
   "repliesIds":[

   ],
   "repliesNames":{

   },
   "repliesColumns":{

   },
   "teamsIds":[
      "609b9a9e5b55802d4"
   ],
   "teamsNames":{
      "609b9a9e5b55802d4":"Obchod ESMERO (všetci)"
   },
   "assignedUsersIds":[
      "630c8dc00dd195da1",
      "63ff12476ca2186d7"
   ],
   "assignedUsersNames":{
      "630c8dc00dd195da1":"Admin Jakub",
      "63ff12476ca2186d7":"Klička Tomáš"
   },
   "createdEventId":null,
   "createdEventType":null,
   "createdEventName":null,
   "groupFolderId":null,
   "groupFolderName":null,
   "accountId":null,
   "accountName":null,
   "tasksIds":[

   ],
   "tasksNames":{

   },
   "tasksColumns":{

   },
   "rodicPrilezitostId":null,
   "rodicPrilezitostType":null,
   "rodicPrilezitostName":null,
   "isFollowed":true,
   "followersIds":[
      "630c8dc00dd195da1",
      "63ff12476ca2186d7"
   ],
   "followersNames":{
      "630c8dc00dd195da1":"Admin Jakub",
      "63ff12476ca2186d7":"Klička Tomáš"
   },
   "bccEmailAddressesIds":[

   ],
   "bccEmailAddressesNames":{

   },
   "attachmentsTypes":{

   }
}

JSON of the duplicate email with status Archived (here you can see there is also switched position of first name and last name in "fromName" etc. :

Code:

{
   "id":"640880e398ff7fabd",
   "name":"ble",
   "deleted":false,
   "subject":"ble",
   "fromName":"Tomáš Klička",
   "fromAddress":"tomas.klicka@***",
   "fromString":"Tomáš Klička <tomas.klicka@***>",
   "replyToString":null,
   "replyToName":null,
   "replyToAddress":null,
   "addressNameMap":{
      "tomas.klicka@***":"Tomáš Klička",
      "jakub.grufik@***":"Jakub Grufik"
   },
   "from":"tomas.klicka@esmero.eu",
   "to":"jakub.grufik@esmero.eu",
   "cc":"",
   "bcc":"",
   "replyTo":"",
   "isRead":true,
   "isReplied":false,
   "isImportant":false,
   "inTrash":false,
   "isUsers":true,
   "nameHash":{
      "tomas.klicka@esmero.eu":"Klička Tomáš",
      "jakub.grufik@esmero.eu":"Admin Jakub"
   },
   "typeHash":{
      "tomas.klicka@esmero.eu":"User",
      "jakub.grufik@esmero.eu":"User"
   },
   "idHash":{
      "tomas.klicka@esmero.eu":"63ff12476ca2186d7",
      "jakub.grufik@esmero.eu":"630c8dc00dd195da1"
   },
   "messageId":"<01020186c0b08b84-f1a5caec-2b2e-4e1b-a196-d0f899350575-000000@eu-west-1.amazonses.com>",
   "messageIdInternal":null,
   "bodyPlain":"bla\r\n\r\n",
   "body":"<p>bla</p>",
   "isHtml":true,
   "status":"Archived",
   "hasAttachment":false,
   "dateSent":"2023-03-08 10:07:20",
   "deliveryDate":null,
   "createdAt":"2023-03-08 12:34:43",
   "modifiedAt":"2023-03-08 12:34:43",
   "isSystem":false,
   "icsContents":null,
   "icsEventUid":null,
   "fromEmailAddressId":"63ff07fa36ea149d4",
   "fromEmailAddressName":"tomas.klicka@***",
   "toEmailAddressesIds":[
      "6363bf5c22f7efaf0"
   ],
   "toEmailAddressesNames":{
      "6363bf5c22f7efaf0":"jakub.grufik@***"
   },
   "ccEmailAddressesIds":[

   ],
   "ccEmailAddressesNames":{

   },
   "replyToEmailAddressesIds":[

   ],
   "replyToEmailAddressesNames":{

   },
   "attachmentsIds":[

   ],
   "attachmentsNames":{

   },
   "parentId":null,
   "parentType":null,
   "parentName":null,
   "createdById":"system",
   "createdByName":"System",
   "sentById":"63ff12476ca2186d7",
   "sentByName":"Klička Tomáš",
   "modifiedById":null,
   "modifiedByName":null,
   "assignedUserId":null,
   "assignedUserName":null,
   "repliedId":null,
   "repliedName":null,
   "repliesIds":[

   ],
   "repliesNames":{

   },
   "repliesColumns":{

   },
   "teamsIds":[
      "609b9a9e5b55802d4"
   ],
   "teamsNames":{
      "609b9a9e5b55802d4":"Obchod ESMERO (všetci)"
   },
   "assignedUsersIds":[
      "630c8dc00dd195da1",
      "63ff12476ca2186d7"
   ],
   "assignedUsersNames":{
      "630c8dc00dd195da1":"Admin Jakub",
      "63ff12476ca2186d7":"Klička Tomáš"
   },
   "createdEventId":null,
   "createdEventType":null,
   "createdEventName":null,
   "groupFolderId":null,
   "groupFolderName":null,
   "accountId":null,
   "accountName":null,
   "tasksIds":[

   ],
   "tasksNames":{

   },
   "tasksColumns":{

   },
   "rodicPrilezitostId":null,
   "rodicPrilezitostType":null,
   "rodicPrilezitostName":null,
   "isFollowed":true,
   "followersIds":[
      "630c8dc00dd195da1",
      "63ff12476ca2186d7"
   ],
   "followersNames":{
      "630c8dc00dd195da1":"Admin Jakub",
      "63ff12476ca2186d7":"Klička Tomáš"
   },
   "bccEmailAddressesIds":[

   ],
   "bccEmailAddressesNames":{

   },
   "attachmentsTypes":{

   }
}

I am sending this just to provide some info until I will be able to get the raw content of the archived email (still trying to manage where to get it as this email is not being stored on the email server). It is present only in CRM.

**yuri** · 04-04-2023, 07:58 AM

Original is not stored in CRM.

**yuri** · 04-04-2023, 09:43 AM

You can send an email from CRM to your personal Gmail/Outlook/any_other account. And get the original from there.

**Jakub Grufik** · 04-04-2023, 11:16 AM

Got it yuri

Here is the raw content of the original email:

Code:

Delivered-To: grufik.k@gmail.com
Received: by 2002:a05:6a10:8681:b0:474:bebc:df90 with SMTP id e1csp2951395pxn;
Tue, 4 Apr 2023 04:08:47 -0700 (PDT)
X-Google-Smtp-Source: AKy350bp2weRdoGxT2dmPCgTYkqAc7F/FEOLh9IWPbKbxDgyzAUUMImS/jOIP/+1bEAJas4ClOOq
X-Received: by 2002:adf:fac3:0:b0:2d9:5608:ee0 with SMTP id a3-20020adffac3000000b002d956080ee0mr975396wrs.69.1680606526736;
Tue, 04 Apr 2023 04:08:46 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1680606526; cv=none;
d=google.com; s=arc-20160816;
b=x+DFD1IeUO5vuYcSgsgeHJqAsG3KUdotzRvQbYMS8sSmmajWXAqIGCz+Dj5nX2Uzm2
oquc2R6j2SimE5PWHbMgZ+3mloVmCvNImUeJF3zf2jEqiAW4oUwQFLdlO2vANdmefPqW
QxPn10yFfcQSWFepHO1Pm6RBtxqQxYuHfPKKvQP4FVTh6cs6Tq0O8/3AIssiy5/0Ver5
HZwwNhJ4bK++t8yybf5YMwkPT2HwbjSM67iYZlPTPX57Rry+MOnyw7pnYVR/YNeBz8/Z
eSM4lHPoOCGYmWfXreq3Ybl7cLIiDUzS5k8Temp+Eza9mLb+xwjplQI1R9rZRjUfnjMi
AWiQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
h=feedback-id:message-id:thread-topic:thread-index:references
:mime-version:date:to:from:subject:dkim-signature:dkim-signature;
bh=0bev6RJvUUQuD2OGslKi4yScwE5GFsQf7Y46VJj0DDo=;
b=VufS7Ck9XbxFiCd0y+UL6Xt9x3jbCW4Rb7YTNqZNKCWvDFZ59qtE14ANsun/LugOHx
E7AQRaF8vXY+WX7OP5fqXZAPtDzbl1srhZrNnYOTeiZTzdNtW0NbMrR+HOXOxtKlqUX9
K4y0mB3BSYG0a6gODA6pUnjdcyymVxCYuCnGVlTD4lz4szyzn36M+x6WQS90ik/gBuDw
jAyxIzyNpjavdWK0A3MCqWV7rzYw4KcLjGFGr3ZLAk+BG0E/sjyBzOnh7ho4tv1fafj8
F3cLl6HB0xQr2zYQuQCB5HRTIe4idjTStuYOAugIKarGDpl59Xd6wPQvo5SBhEhu2iHN
augA==
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=@esmero.eu header.s=vr4lk5ymt4znqsinu7p6tldo3p7npuxk header.b=DlPV4jLU;
dkim=pass header.i=@amazonses.com header.s=shh3fegwg5fppqsuzphvschd53n6ihuv header.b=eAZNBHqp;
spf=pass (google.com: domain of 010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com designates 54.240.2.78 as permitted sender) smtp.mailfrom=010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=esmero.eu
Return-Path: <010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com>
Received: from a2-78.smtp-out.eu-west-1.amazonses.com (a2-78.smtp-out.eu-west-1.amazonses.com. [54.240.2.78])
by mx.google.com with ESMTPS id d3-20020adffd83000000b002c406802ab8si5090796wrr.596.2023.04.04.04.08.44
for <grufik.k@gmail.com>
(version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
Tue, 04 Apr 2023 04:08:46 -0700 (PDT)
Received-SPF: pass (google.com: domain of 010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com designates 54.240.2.78 as permitted sender) client-ip=54.240.2.78;
Authentication-Results: mx.google.com;
dkim=pass header.i=@esmero.eu header.s=vr4lk5ymt4znqsinu7p6tldo3p7npuxk header.b=DlPV4jLU;
dkim=pass header.i=@amazonses.com header.s=shh3fegwg5fppqsuzphvschd53n6ihuv header.b=eAZNBHqp;
spf=pass (google.com: domain of 010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com designates 54.240.2.78 as permitted sender) smtp.mailfrom=010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=esmero.eu
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=vr4lk5ymt4znqsinu7p6tldo3p7npuxk; d=esmero.eu; t=1680606524; h=Subject:From:To:Date:Mime-Version:Content-Type:References:Message-Id; bh=0bev6RJvUUQuD2OGslKi4yScwE5GFsQf7Y46VJj0DDo=; b=DlPV4jLUViMtLGUyDxEXe4NWORA3OipMumm+ks88d9RxQ9Yt0/BTg7qCoQwCg9pH iRE6P/co9n4diiTeYJewlrftOsU489NdnhYh0ElrR0YKmJswYy8fmQVJM0ItyYIZhTf KTGl08EDxkBlsNk7HZ0RfLyi7W3LD21Pk2P+Hxcqf3z+2QTS9bvtmgBkpAvHhuB7G3x IekyPQR2gGPHWDKOpBXZA+1Fv7xg/UW3JBffS1hcueQJ/43HaqFKpCUNxw1hVtkoTrp yyaA5gCYKAp3KBZ8KoQfvOoUUZiY+/W6GdxSsroz3OlVR+Ss5s9D9NthIdMqaWKv49R HulyjwJswg==
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=shh3fegwg5fppqsuzphvschd53n6ihuv; d=amazonses.com; t=1680606524; h=Subject:From:To:Date:Mime-Version:Content-Type:References:Message-Id:Feedback-ID; bh=0bev6RJvUUQuD2OGslKi4yScwE5GFsQf7Y46VJj0DDo=; b=eAZNBHqp/uTpayfapMU1AtLInInCk+0k99FwVRvUq4K8HgJEfYOlTSbI3znr3/uR j0felBp74gup8Ra0IcHxNnc5knn6pJLkKgMIDlkK41T2iB55wf1oVji1i8YSWMQkcKB OhrOgUr2zMfF5V+ILtK0h0G/4OAyNm+zlqeeda00=
Subject: Test Duplicate
From: Jakub Grufik <jakub.grufik@esmero.eu>
To: "grufik.k@gmail.com" <grufik.k@gmail.com>
Date: Tue, 4 Apr 2023 11:08:44 +0000
Mime-Version: 1.0
Content-Type: multipart/alternative;
boundary="=_Xk0bnbO22557egI6zcnflXvkP7lc5yO-3fj8SskPXEy+ZtEx"
References: <3875b49502182d7d470e02efc212a176/1680606520/8170@espo>
X-Mailer: Amazon WorkMail
Thread-Index: AQHZZuXQLMbunL+vRAacgFdTqWh5sQ==
Thread-Topic: Test Duplicate
X-Wm-Sent-Timestamp: 1680606523
Message-ID: <010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com>
Feedback-ID: 1.eu-west-1.b24dn6frgCi6dh20skzbuMRr7UL8M6Soir/3ogtEjHQ=:AmazonSES
X-SES-Outgoing: 2023.04.04-54.240.2.78

--=_Xk0bnbO22557egI6zcnflXvkP7lc5yO-3fj8SskPXEy+ZtEx
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

S pozdravem,

Jakub Gruf=C3=ADk
V=C3=BDvoj=C3=A1=C5=99 CRM

tel.:=C2=A0+420=C2=A0725 146 100
web:=C2=A0www.esmerocz.eu
mail.:=C2=A0jakub.grufik@esmerocz.cz
fb.:=C2=A0esmerozimnizahrady
instagram:=C2=A0esmerocz
youtube:=C2=A0esmero


--=_Xk0bnbO22557egI6zcnflXvkP7lc5yO-3fj8SskPXEy+ZtEx
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64

The original message ID is stored in the attribute "References"

Attached Files

**yuri** · 04-04-2023, 11:40 AM

Not good. This would be tricky to distinguish from a legit reply email. As the references header is supposed to contain the ID of the email the current email is replied to.

**Jakub Grufik** · 04-21-2023, 07:22 AM

Hello guys, can someone help me to solve this issue, please? We have a looot of unnecessary emails in our system and basically, everything is being duplicated with status "Archived".

I dont think this is caused by the fact that amazon is changing the ID of the message. In that case it would duplicate everything with status "Sent" or not? One email has status "Sent" and the duplicate has status "Archived". Why is this happening please

Our system is srsly overloaded with emails atm.

Email duplicates created by system

Email duplicates created by system

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment