Email duplicates created by system

[yuri]

I see that I already provided the code https://forum.espocrm.com/forum/gene...4917#post84917

Try it with 'References' instead of 'X-Microsoft-Original-Message-ID'. But I recommend to find a developer who will support it for a fee.

[yuri]

A user sends email from CRM. It's stored with its original message ID.
SMTP sends this email with different message ID.
CRM monitors another mailbox that receives that email.
The email is imported as it's not detected as already existing in the system.

[Jakub Grufik]

hey yuri I understand that you are overwhelmed with tasks. But is it really connected with duplicate check when the status is different? Original email has status Sent and duplicate is status Archived. Could not be somewhere in the system hidden that it is auto-archiving every email?

I am just confused why everything is being archived.

[yuri]

I suggest finding a developer who will customize the duplicate checking for you for a fee.

I'm not able to take more obligations on me. I'm handing dozens of issues daily.

[Jakub Grufik]

Hello guys, can someone help me to solve this issue, please? We have a looot of unnecessary emails in our system and basically, everything is being duplicated with status "Archived".

I dont think this is caused by the fact that amazon is changing the ID of the message. In that case it would duplicate everything with status "Sent" or not? One email has status "Sent" and the duplicate has status "Archived". Why is this happening please

Our system is srsly overloaded with emails atm.

[Jakub Grufik]

can it be somewhere in the settings of the project that every email is being archived automaticaly? I tried to look for it but without success.. for me it would make sense that problem with duplicates would be if both emails have status "sent" for example..but one is sent the other one is archived, so I do not know why the system is archiving every single email..

[yuri]

Not good. This would be tricky to distinguish from a legit reply email. As the references header is supposed to contain the ID of the email the current email is replied to.

[Jakub Grufik]

Got it yuri

Here is the raw content of the original email:

Delivered-To: grufik.k@gmail.com
Received: by 2002:a05:6a10:8681:b0:474:bebc:df90 with SMTP id e1csp2951395pxn;
Tue, 4 Apr 2023 04:08:47 -0700 (PDT)
X-Google-Smtp-Source: AKy350bp2weRdoGxT2dmPCgTYkqAc7F/FEOLh9IWPbKbxDgyzAUUMImS/jOIP/+1bEAJas4ClOOq
X-Received: by 2002:adf:fac3:0:b0:2d9:5608:ee0 with SMTP id a3-20020adffac3000000b002d956080ee0mr975396wrs.69.1680606526736;
Tue, 04 Apr 2023 04:08:46 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1680606526; cv=none;
d=google.com; s=arc-20160816;
b=x+DFD1IeUO5vuYcSgsgeHJqAsG3KUdotzRvQbYMS8sSmmajWXAqIGCz+Dj5nX2Uzm2
oquc2R6j2SimE5PWHbMgZ+3mloVmCvNImUeJF3zf2jEqiAW4oUwQFLdlO2vANdmefPqW
QxPn10yFfcQSWFepHO1Pm6RBtxqQxYuHfPKKvQP4FVTh6cs6Tq0O8/3AIssiy5/0Ver5
HZwwNhJ4bK++t8yybf5YMwkPT2HwbjSM67iYZlPTPX57Rry+MOnyw7pnYVR/YNeBz8/Z
eSM4lHPoOCGYmWfXreq3Ybl7cLIiDUzS5k8Temp+Eza9mLb+xwjplQI1R9rZRjUfnjMi
AWiQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
h=feedback-id:message-id:thread-topic:thread-index:references
:mime-version:date:to:from:subject:dkim-signature:dkim-signature;
bh=0bev6RJvUUQuD2OGslKi4yScwE5GFsQf7Y46VJj0DDo=;
b=VufS7Ck9XbxFiCd0y+UL6Xt9x3jbCW4Rb7YTNqZNKCWvDFZ59qtE14ANsun/LugOHx
E7AQRaF8vXY+WX7OP5fqXZAPtDzbl1srhZrNnYOTeiZTzdNtW0NbMrR+HOXOxtKlqUX9
K4y0mB3BSYG0a6gODA6pUnjdcyymVxCYuCnGVlTD4lz4szyzn36M+x6WQS90ik/gBuDw
jAyxIzyNpjavdWK0A3MCqWV7rzYw4KcLjGFGr3ZLAk+BG0E/sjyBzOnh7ho4tv1fafj8
F3cLl6HB0xQr2zYQuQCB5HRTIe4idjTStuYOAugIKarGDpl59Xd6wPQvo5SBhEhu2iHN
augA==
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=@esmero.eu header.s=vr4lk5ymt4znqsinu7p6tldo3p7npuxk header.b=DlPV4jLU;
dkim=pass header.i=@amazonses.com header.s=shh3fegwg5fppqsuzphvschd53n6ihuv header.b=eAZNBHqp;
spf=pass (google.com: domain of 010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com designates 54.240.2.78 as permitted sender) smtp.mailfrom=010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=esmero.eu
Return-Path: <010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com>
Received: from a2-78.smtp-out.eu-west-1.amazonses.com (a2-78.smtp-out.eu-west-1.amazonses.com. [54.240.2.78])
by mx.google.com with ESMTPS id d3-20020adffd83000000b002c406802ab8si5090796wrr.596.2023.04.04.04.08.44
for <grufik.k@gmail.com>
(version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
Tue, 04 Apr 2023 04:08:46 -0700 (PDT)
Received-SPF: pass (google.com: domain of 010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com designates 54.240.2.78 as permitted sender) client-ip=54.240.2.78;
Authentication-Results: mx.google.com;
dkim=pass header.i=@esmero.eu header.s=vr4lk5ymt4znqsinu7p6tldo3p7npuxk header.b=DlPV4jLU;
dkim=pass header.i=@amazonses.com header.s=shh3fegwg5fppqsuzphvschd53n6ihuv header.b=eAZNBHqp;
spf=pass (google.com: domain of 010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com designates 54.240.2.78 as permitted sender) smtp.mailfrom=010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=esmero.eu
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=vr4lk5ymt4znqsinu7p6tldo3p7npuxk; d=esmero.eu; t=1680606524; h=Subject:From:To:Date:Mime-Version:Content-Type:References:Message-Id; bh=0bev6RJvUUQuD2OGslKi4yScwE5GFsQf7Y46VJj0DDo=; b=DlPV4jLUViMtLGUyDxEXe4NWORA3OipMumm+ks88d9RxQ9Yt0/BTg7qCoQwCg9pH iRE6P/co9n4diiTeYJewlrftOsU489NdnhYh0ElrR0YKmJswYy8fmQVJM0ItyYIZhTf KTGl08EDxkBlsNk7HZ0RfLyi7W3LD21Pk2P+Hxcqf3z+2QTS9bvtmgBkpAvHhuB7G3x IekyPQR2gGPHWDKOpBXZA+1Fv7xg/UW3JBffS1hcueQJ/43HaqFKpCUNxw1hVtkoTrp yyaA5gCYKAp3KBZ8KoQfvOoUUZiY+/W6GdxSsroz3OlVR+Ss5s9D9NthIdMqaWKv49R HulyjwJswg==
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=shh3fegwg5fppqsuzphvschd53n6ihuv; d=amazonses.com; t=1680606524; h=Subject:From:To:Date:Mime-Version:Content-Type:References:Message-Id:Feedback-ID; bh=0bev6RJvUUQuD2OGslKi4yScwE5GFsQf7Y46VJj0DDo=; b=eAZNBHqp/uTpayfapMU1AtLInInCk+0k99FwVRvUq4K8HgJEfYOlTSbI3znr3/uR j0felBp74gup8Ra0IcHxNnc5knn6pJLkKgMIDlkK41T2iB55wf1oVji1i8YSWMQkcKB OhrOgUr2zMfF5V+ILtK0h0G/4OAyNm+zlqeeda00=
Subject: Test Duplicate
From: Jakub Grufik <jakub.grufik@esmero.eu>
To: "grufik.k@gmail.com" <grufik.k@gmail.com>
Date: Tue, 4 Apr 2023 11:08:44 +0000
Mime-Version: 1.0
Content-Type: multipart/alternative;
boundary="=_Xk0bnbO22557egI6zcnflXvkP7lc5yO-3fj8SskPXEy+ZtEx"
References: <3875b49502182d7d470e02efc212a176/1680606520/8170@espo>
X-Mailer: Amazon WorkMail
Thread-Index: AQHZZuXQLMbunL+vRAacgFdTqWh5sQ==
Thread-Topic: Test Duplicate
X-Wm-Sent-Timestamp: 1680606523
Message-ID: <010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com>
Feedback-ID: 1.eu-west-1.b24dn6frgCi6dh20skzbuMRr7UL8M6Soir/3ogtEjHQ=:AmazonSES
X-SES-Outgoing: 2023.04.04-54.240.2.78

--=_Xk0bnbO22557egI6zcnflXvkP7lc5yO-3fj8SskPXEy+ZtEx
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

S pozdravem,

Jakub Gruf=C3=ADk
V=C3=BDvoj=C3=A1=C5=99 CRM

tel.:=C2=A0+420=C2=A0725 146 100
web:=C2=A0www.esmerocz.eu
mail.:=C2=A0jakub.grufik@esmerocz.cz
fb.:=C2=A0esmerozimnizahrady
instagram:=C2=A0esmerocz
youtube:=C2=A0esmero


--=_Xk0bnbO22557egI6zcnflXvkP7lc5yO-3fj8SskPXEy+ZtEx
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64​

The original message ID is stored in the attribute "References"

[yuri]

You can send an email from CRM to your personal Gmail/Outlook/any_other account. And get the original from there.

[Jakub Grufik]

seems like I do not have access to the email database of the email provider to be able to get raw content of the original message.

[Jakub Grufik]

I am not sure where to find it, because the duplicate email is not being shown on the email server.. Will try to dig more into it..

[yuri]

Original is not stored in CRM.

[Jakub Grufik]

Yes, seems like smtp is changing the message id:

JSON of email with status sent:

{
   "id":"64085e55a2c6a5bc4",
   "name":"ble",
   "deleted":false,
   "subject":"ble",
   "fromName":"Klička Tomáš",
   "fromAddress":"tomas.klicka@***",
   "fromString":"Klička Tomáš <tomas.klicka@***>",
   "replyToString":null,
   "replyToName":null,
   "replyToAddress":null,
   "addressNameMap":null,
   "from":"tomas.klicka@***",
   "to":"jakub.grufik@***",
   "cc":"",
   "bcc":"",
   "replyTo":"",
   "isRead":true,
   "isReplied":false,
   "isImportant":false,
   "inTrash":false,
   "isUsers":true,
   "nameHash":{
      "tomas.klicka***":"Klička Tomáš",
      "jakub.grufik@***":"Admin Jakub"
   },
   "typeHash":{
      "tomas.klicka@***":"User",
      "jakub.grufik@***":"User"
   },
   "idHash":{
      "tomas.klicka@***":"63ff12476ca2186d7",
      "jakub.grufik@***":"630c8dc00dd195da1"
   },
   "messageId":"<4e8a1f3702ea40975a6bd7b06e558498/1678270037/1848@espo>",
   "messageIdInternal":null,
   "bodyPlain":"bla\n\n",
   "body":"<p>bla</p>",
   "isHtml":true,
   "status":"Sent",
   "hasAttachment":false,
   "dateSent":"2023-03-08 10:07:19",
   "deliveryDate":null,
   "createdAt":"2023-03-08 10:07:17",
   "modifiedAt":"2023-03-08 10:07:19",
   "isSystem":false,
   "icsContents":null,
   "icsEventUid":null,
   "fromEmailAddressId":"63ff07fa36ea149d4",
   "fromEmailAddressName":"tomas.klicka@***",
   "toEmailAddressesIds":[
      "6363bf5c22f7efaf0"
   ],
   "toEmailAddressesNames":{
      "6363bf5c22f7efaf0":"jakub.grufik@***"
   },
   "ccEmailAddressesIds":[

   ],
   "ccEmailAddressesNames":{

   },
   "replyToEmailAddressesIds":[

   ],
   "replyToEmailAddressesNames":{

   },
   "attachmentsIds":[

   ],
   "attachmentsNames":{

   },
   "parentId":null,
   "parentType":null,
   "parentName":null,
   "createdById":"63ff12476ca2186d7",
   "createdByName":"Klička Tomáš",
   "sentById":"63ff12476ca2186d7",
   "sentByName":"Klička Tomáš",
   "modifiedById":"63ff12476ca2186d7",
   "modifiedByName":"Klička Tomáš",
   "assignedUserId":null,
   "assignedUserName":null,
   "repliedId":null,
   "repliedName":null,
   "repliesIds":[

   ],
   "repliesNames":{

   },
   "repliesColumns":{

   },
   "teamsIds":[
      "609b9a9e5b55802d4"
   ],
   "teamsNames":{
      "609b9a9e5b55802d4":"Obchod ESMERO (všetci)"
   },
   "assignedUsersIds":[
      "630c8dc00dd195da1",
      "63ff12476ca2186d7"
   ],
   "assignedUsersNames":{
      "630c8dc00dd195da1":"Admin Jakub",
      "63ff12476ca2186d7":"Klička Tomáš"
   },
   "createdEventId":null,
   "createdEventType":null,
   "createdEventName":null,
   "groupFolderId":null,
   "groupFolderName":null,
   "accountId":null,
   "accountName":null,
   "tasksIds":[

   ],
   "tasksNames":{

   },
   "tasksColumns":{

   },
   "rodicPrilezitostId":null,
   "rodicPrilezitostType":null,
   "rodicPrilezitostName":null,
   "isFollowed":true,
   "followersIds":[
      "630c8dc00dd195da1",
      "63ff12476ca2186d7"
   ],
   "followersNames":{
      "630c8dc00dd195da1":"Admin Jakub",
      "63ff12476ca2186d7":"Klička Tomáš"
   },
   "bccEmailAddressesIds":[

   ],
   "bccEmailAddressesNames":{

   },
   "attachmentsTypes":{

   }
}​

JSON of the duplicate email with status Archived (here you can see there is also switched position of first name and last name in "fromName" etc. :

{
   "id":"640880e398ff7fabd",
   "name":"ble",
   "deleted":false,
   "subject":"ble",
   "fromName":"Tomáš Klička",
   "fromAddress":"tomas.klicka@***",
   "fromString":"Tomáš Klička <tomas.klicka@***>",
   "replyToString":null,
   "replyToName":null,
   "replyToAddress":null,
   "addressNameMap":{
      "tomas.klicka@***":"Tomáš Klička",
      "jakub.grufik@***":"Jakub Grufik"
   },
   "from":"tomas.klicka@esmero.eu",
   "to":"jakub.grufik@esmero.eu",
   "cc":"",
   "bcc":"",
   "replyTo":"",
   "isRead":true,
   "isReplied":false,
   "isImportant":false,
   "inTrash":false,
   "isUsers":true,
   "nameHash":{
      "tomas.klicka@esmero.eu":"Klička Tomáš",
      "jakub.grufik@esmero.eu":"Admin Jakub"
   },
   "typeHash":{
      "tomas.klicka@esmero.eu":"User",
      "jakub.grufik@esmero.eu":"User"
   },
   "idHash":{
      "tomas.klicka@esmero.eu":"63ff12476ca2186d7",
      "jakub.grufik@esmero.eu":"630c8dc00dd195da1"
   },
   "messageId":"<01020186c0b08b84-f1a5caec-2b2e-4e1b-a196-d0f899350575-000000@eu-west-1.amazonses.com>",
   "messageIdInternal":null,
   "bodyPlain":"bla\r\n\r\n",
   "body":"<p>bla</p>",
   "isHtml":true,
   "status":"Archived",
   "hasAttachment":false,
   "dateSent":"2023-03-08 10:07:20",
   "deliveryDate":null,
   "createdAt":"2023-03-08 12:34:43",
   "modifiedAt":"2023-03-08 12:34:43",
   "isSystem":false,
   "icsContents":null,
   "icsEventUid":null,
   "fromEmailAddressId":"63ff07fa36ea149d4",
   "fromEmailAddressName":"tomas.klicka@***",
   "toEmailAddressesIds":[
      "6363bf5c22f7efaf0"
   ],
   "toEmailAddressesNames":{
      "6363bf5c22f7efaf0":"jakub.grufik@***"
   },
   "ccEmailAddressesIds":[

   ],
   "ccEmailAddressesNames":{

   },
   "replyToEmailAddressesIds":[

   ],
   "replyToEmailAddressesNames":{

   },
   "attachmentsIds":[

   ],
   "attachmentsNames":{

   },
   "parentId":null,
   "parentType":null,
   "parentName":null,
   "createdById":"system",
   "createdByName":"System",
   "sentById":"63ff12476ca2186d7",
   "sentByName":"Klička Tomáš",
   "modifiedById":null,
   "modifiedByName":null,
   "assignedUserId":null,
   "assignedUserName":null,
   "repliedId":null,
   "repliedName":null,
   "repliesIds":[

   ],
   "repliesNames":{

   },
   "repliesColumns":{

   },
   "teamsIds":[
      "609b9a9e5b55802d4"
   ],
   "teamsNames":{
      "609b9a9e5b55802d4":"Obchod ESMERO (všetci)"
   },
   "assignedUsersIds":[
      "630c8dc00dd195da1",
      "63ff12476ca2186d7"
   ],
   "assignedUsersNames":{
      "630c8dc00dd195da1":"Admin Jakub",
      "63ff12476ca2186d7":"Klička Tomáš"
   },
   "createdEventId":null,
   "createdEventType":null,
   "createdEventName":null,
   "groupFolderId":null,
   "groupFolderName":null,
   "accountId":null,
   "accountName":null,
   "tasksIds":[

   ],
   "tasksNames":{

   },
   "tasksColumns":{

   },
   "rodicPrilezitostId":null,
   "rodicPrilezitostType":null,
   "rodicPrilezitostName":null,
   "isFollowed":true,
   "followersIds":[
      "630c8dc00dd195da1",
      "63ff12476ca2186d7"
   ],
   "followersNames":{
      "630c8dc00dd195da1":"Admin Jakub",
      "63ff12476ca2186d7":"Klička Tomáš"
   },
   "bccEmailAddressesIds":[

   ],
   "bccEmailAddressesNames":{

   },
   "attachmentsTypes":{

   }
}​

I am sending this just to provide some info until I will be able to get the raw content of the archived email (still trying to manage where to get it as this email is not being stored on the email server). It is present only in CRM.

[yuri]

Can you provide an email original (raw content) of a duplicate email?

Looks like your SMTP is changing the message ID. Maybe the original message-ID is preserved in some header.

[Jakub Grufik]

Current version: 7.3.3