Announcement

Collapse
No announcement yet.

Email duplicates created by system

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Email duplicates created by system

    Hello,

    yesterday we changed our email provider to amazon work mail.

    Since we have changed it, we are facing issues that when we send an email, a duplicate is created by the system. This is happening with both inbound and outbound mails.

    We deleted the original Personal email account and added the new one. Also, the email address of each user has been rewritten in their account, and then Cache was cleared.

    Do you guys have any idea what could be wrong, please?
    When someone sends an email to me, I got it twice in my inbox, 1 created by the sender, and 1 created by the system.
    Also when I send an email, I have it shown twice in my "sent" folder.

    Thanks a lot for the help!

    edit: This is happening only when we are sending emails from CRM to CRM, it is not creating duplicates when we send emails from Outlook to CRM for example.
    Attached Files
    Last edited by Jakub Grufik; 11-09-2022, 07:42 AM.

  • #2
    Hi Jakub,

    Check what message-IDs imported duplicates have (table email > column message_id). It could be that your SMTP (or maybe IMAP?) changes message IDs and Espo can't detect that it's a duplicate. Espo checks duplicates only by message-ID.

    Comment


    • #3
      There will be the ability to bind a custom implementation of the duplicate finder in the next version.

      https://github.com/espocrm/espocrm/b...nding.php#L249

      https://github.com/espocrm/espocrm/b...cateFinder.php

      Comment


      • #4
        Likely, that SMTP adds a header with an original message-ID (Microsoft adds X-Microsoft-Original-Message-ID header). The idea is to check this ID too.

        PHP Code:
            public function find(Email $emailMessage $message): ?Email
            
        {
                $messageIds 
        = [];

                if (
        $email->getMessageId()) {
                    
        $messageIds[] = $email->getMessageId();
                }

                if (
        $message->getHeader('X-Microsoft-Original-Message-ID')) {
                    
        $messageIds[] = $message->getHeader('X-Microsoft-Original-Message-ID');
                }

                if (
        $messageIds === []) {
                    return 
        null;    
                }


                
        return $this->entityManager
                    
        ->getRDBRepositoryByClass(Email::class)
                    ->
        select(['id''status'])
                    ->
        where([
                        
        'messageId' => $messageIds,
                    ])
                    ->
        findOne();
            


        But this will be possible only in v7.3.

        Comment


        • #5
          Hello,
          yeah both emails have different message_id.

          I am not sure if I understand correctly, but at the moment we are not able to fix this unless the email provider stops changing the ids of the message(that is probably not going to happen), correct?

          Comment


          • #6
            Correct. Without making modifications to code (that won't work after upgrade) there's no way. But as of v7.3 there's a way. Can you check a message original (raw contents of an email) of any of such emails (in any email client or web interface of your mail provider). Whether it contains an original message ID header.

            Comment


            • #7
              Hey yuri thanks a lot for the explanation and information.

              Will pass this info to our email administrator to find out what could be done about this behavior.

              Have a nice day sir!

              Comment


              • #8
                Hello guys, I would like to re-open this topic as our system is still being overloaded by duplicates of every single email sent within CRM.

                If it goes from outlook or Gmail etc. to CRM there is no duplicate. But when we send an email from CRM to CRM (internally), a duplicate with the status "Archived" is created every single time.

                Is there already any way how to solve this behavior, please?

                Thanks a lot

                Comment


                • Jakub Grufik
                  Jakub Grufik commented
                  Editing a comment
                  Current version: 7.3.3

              • #9
                Can you provide an email original (raw content) of a duplicate email?

                Looks like your SMTP is changing the message ID. Maybe the original message-ID is preserved in some header.

                Comment


                • #10
                  Yes, seems like smtp is changing the message id:

                  JSON of email with status sent:
                  Code:
                  {
                     "id":"64085e55a2c6a5bc4",
                     "name":"ble",
                     "deleted":false,
                     "subject":"ble",
                     "fromName":"Klička Tomáš",
                     "fromAddress":"tomas.klicka@***",
                     "fromString":"Klička Tomáš <tomas.klicka@***>",
                     "replyToString":null,
                     "replyToName":null,
                     "replyToAddress":null,
                     "addressNameMap":null,
                     "from":"tomas.klicka@***",
                     "to":"jakub.grufik@***",
                     "cc":"",
                     "bcc":"",
                     "replyTo":"",
                     "isRead":true,
                     "isReplied":false,
                     "isImportant":false,
                     "inTrash":false,
                     "isUsers":true,
                     "nameHash":{
                        "tomas.klicka***":"Klička Tomáš",
                        "jakub.grufik@***":"Admin Jakub"
                     },
                     "typeHash":{
                        "tomas.klicka@***":"User",
                        "jakub.grufik@***":"User"
                     },
                     "idHash":{
                        "tomas.klicka@***":"63ff12476ca2186d7",
                        "jakub.grufik@***":"630c8dc00dd195da1"
                     },
                     "messageId":"<4e8a1f3702ea40975a6bd7b06e558498/1678270037/1848@espo>",
                     "messageIdInternal":null,
                     "bodyPlain":"bla\n\n",
                     "body":"<p>bla</p>",
                     "isHtml":true,
                     "status":"Sent",
                     "hasAttachment":false,
                     "dateSent":"2023-03-08 10:07:19",
                     "deliveryDate":null,
                     "createdAt":"2023-03-08 10:07:17",
                     "modifiedAt":"2023-03-08 10:07:19",
                     "isSystem":false,
                     "icsContents":null,
                     "icsEventUid":null,
                     "fromEmailAddressId":"63ff07fa36ea149d4",
                     "fromEmailAddressName":"tomas.klicka@***",
                     "toEmailAddressesIds":[
                        "6363bf5c22f7efaf0"
                     ],
                     "toEmailAddressesNames":{
                        "6363bf5c22f7efaf0":"jakub.grufik@***"
                     },
                     "ccEmailAddressesIds":[
                  
                     ],
                     "ccEmailAddressesNames":{
                  
                     },
                     "replyToEmailAddressesIds":[
                  
                     ],
                     "replyToEmailAddressesNames":{
                  
                     },
                     "attachmentsIds":[
                  
                     ],
                     "attachmentsNames":{
                  
                     },
                     "parentId":null,
                     "parentType":null,
                     "parentName":null,
                     "createdById":"63ff12476ca2186d7",
                     "createdByName":"Klička Tomáš",
                     "sentById":"63ff12476ca2186d7",
                     "sentByName":"Klička Tomáš",
                     "modifiedById":"63ff12476ca2186d7",
                     "modifiedByName":"Klička Tomáš",
                     "assignedUserId":null,
                     "assignedUserName":null,
                     "repliedId":null,
                     "repliedName":null,
                     "repliesIds":[
                  
                     ],
                     "repliesNames":{
                  
                     },
                     "repliesColumns":{
                  
                     },
                     "teamsIds":[
                        "609b9a9e5b55802d4"
                     ],
                     "teamsNames":{
                        "609b9a9e5b55802d4":"Obchod ESMERO (všetci)"
                     },
                     "assignedUsersIds":[
                        "630c8dc00dd195da1",
                        "63ff12476ca2186d7"
                     ],
                     "assignedUsersNames":{
                        "630c8dc00dd195da1":"Admin Jakub",
                        "63ff12476ca2186d7":"Klička Tomáš"
                     },
                     "createdEventId":null,
                     "createdEventType":null,
                     "createdEventName":null,
                     "groupFolderId":null,
                     "groupFolderName":null,
                     "accountId":null,
                     "accountName":null,
                     "tasksIds":[
                  
                     ],
                     "tasksNames":{
                  
                     },
                     "tasksColumns":{
                  
                     },
                     "rodicPrilezitostId":null,
                     "rodicPrilezitostType":null,
                     "rodicPrilezitostName":null,
                     "isFollowed":true,
                     "followersIds":[
                        "630c8dc00dd195da1",
                        "63ff12476ca2186d7"
                     ],
                     "followersNames":{
                        "630c8dc00dd195da1":"Admin Jakub",
                        "63ff12476ca2186d7":"Klička Tomáš"
                     },
                     "bccEmailAddressesIds":[
                  
                     ],
                     "bccEmailAddressesNames":{
                  
                     },
                     "attachmentsTypes":{
                  
                     }
                  }​
                  JSON of the duplicate email with status Archived (here you can see there is also switched position of first name and last name in "fromName" etc. :

                  Code:
                  {
                     "id":"640880e398ff7fabd",
                     "name":"ble",
                     "deleted":false,
                     "subject":"ble",
                     "fromName":"Tomáš Klička",
                     "fromAddress":"tomas.klicka@***",
                     "fromString":"Tomáš Klička <tomas.klicka@***>",
                     "replyToString":null,
                     "replyToName":null,
                     "replyToAddress":null,
                     "addressNameMap":{
                        "tomas.klicka@***":"Tomáš Klička",
                        "jakub.grufik@***":"Jakub Grufik"
                     },
                     "from":"tomas.klicka@esmero.eu",
                     "to":"jakub.grufik@esmero.eu",
                     "cc":"",
                     "bcc":"",
                     "replyTo":"",
                     "isRead":true,
                     "isReplied":false,
                     "isImportant":false,
                     "inTrash":false,
                     "isUsers":true,
                     "nameHash":{
                        "tomas.klicka@esmero.eu":"Klička Tomáš",
                        "jakub.grufik@esmero.eu":"Admin Jakub"
                     },
                     "typeHash":{
                        "tomas.klicka@esmero.eu":"User",
                        "jakub.grufik@esmero.eu":"User"
                     },
                     "idHash":{
                        "tomas.klicka@esmero.eu":"63ff12476ca2186d7",
                        "jakub.grufik@esmero.eu":"630c8dc00dd195da1"
                     },
                     "messageId":"<01020186c0b08b84-f1a5caec-2b2e-4e1b-a196-d0f899350575-000000@eu-west-1.amazonses.com>",
                     "messageIdInternal":null,
                     "bodyPlain":"bla\r\n\r\n",
                     "body":"<p>bla</p>",
                     "isHtml":true,
                     "status":"Archived",
                     "hasAttachment":false,
                     "dateSent":"2023-03-08 10:07:20",
                     "deliveryDate":null,
                     "createdAt":"2023-03-08 12:34:43",
                     "modifiedAt":"2023-03-08 12:34:43",
                     "isSystem":false,
                     "icsContents":null,
                     "icsEventUid":null,
                     "fromEmailAddressId":"63ff07fa36ea149d4",
                     "fromEmailAddressName":"tomas.klicka@***",
                     "toEmailAddressesIds":[
                        "6363bf5c22f7efaf0"
                     ],
                     "toEmailAddressesNames":{
                        "6363bf5c22f7efaf0":"jakub.grufik@***"
                     },
                     "ccEmailAddressesIds":[
                  
                     ],
                     "ccEmailAddressesNames":{
                  
                     },
                     "replyToEmailAddressesIds":[
                  
                     ],
                     "replyToEmailAddressesNames":{
                  
                     },
                     "attachmentsIds":[
                  
                     ],
                     "attachmentsNames":{
                  
                     },
                     "parentId":null,
                     "parentType":null,
                     "parentName":null,
                     "createdById":"system",
                     "createdByName":"System",
                     "sentById":"63ff12476ca2186d7",
                     "sentByName":"Klička Tomáš",
                     "modifiedById":null,
                     "modifiedByName":null,
                     "assignedUserId":null,
                     "assignedUserName":null,
                     "repliedId":null,
                     "repliedName":null,
                     "repliesIds":[
                  
                     ],
                     "repliesNames":{
                  
                     },
                     "repliesColumns":{
                  
                     },
                     "teamsIds":[
                        "609b9a9e5b55802d4"
                     ],
                     "teamsNames":{
                        "609b9a9e5b55802d4":"Obchod ESMERO (všetci)"
                     },
                     "assignedUsersIds":[
                        "630c8dc00dd195da1",
                        "63ff12476ca2186d7"
                     ],
                     "assignedUsersNames":{
                        "630c8dc00dd195da1":"Admin Jakub",
                        "63ff12476ca2186d7":"Klička Tomáš"
                     },
                     "createdEventId":null,
                     "createdEventType":null,
                     "createdEventName":null,
                     "groupFolderId":null,
                     "groupFolderName":null,
                     "accountId":null,
                     "accountName":null,
                     "tasksIds":[
                  
                     ],
                     "tasksNames":{
                  
                     },
                     "tasksColumns":{
                  
                     },
                     "rodicPrilezitostId":null,
                     "rodicPrilezitostType":null,
                     "rodicPrilezitostName":null,
                     "isFollowed":true,
                     "followersIds":[
                        "630c8dc00dd195da1",
                        "63ff12476ca2186d7"
                     ],
                     "followersNames":{
                        "630c8dc00dd195da1":"Admin Jakub",
                        "63ff12476ca2186d7":"Klička Tomáš"
                     },
                     "bccEmailAddressesIds":[
                  
                     ],
                     "bccEmailAddressesNames":{
                  
                     },
                     "attachmentsTypes":{
                  
                     }
                  }​
                  I am sending this just to provide some info until I will be able to get the raw content of the archived email (still trying to manage where to get it as this email is not being stored on the email server). It is present only in CRM.
                  Last edited by yuri; 04-04-2023, 08:00 AM.

                  Comment


                  • #11
                    Original is not stored in CRM.

                    Comment


                    • Jakub Grufik
                      Jakub Grufik commented
                      Editing a comment
                      I am not sure where to find it, because the duplicate email is not being shown on the email server.. Will try to dig more into it..

                    • Jakub Grufik
                      Jakub Grufik commented
                      Editing a comment
                      seems like I do not have access to the email database of the email provider to be able to get raw content of the original message.

                  • #12
                    You can send an email from CRM to your personal Gmail/Outlook/any_other account. And get the original from there.

                    Comment


                    • #13
                      Got it yuri

                      Here is the raw content of the original email:
                      Code:
                      Delivered-To: grufik.k@gmail.com
                      Received: by 2002:a05:6a10:8681:b0:474:bebc:df90 with SMTP id e1csp2951395pxn;
                      Tue, 4 Apr 2023 04:08:47 -0700 (PDT)
                      X-Google-Smtp-Source: AKy350bp2weRdoGxT2dmPCgTYkqAc7F/FEOLh9IWPbKbxDgyzAUUMImS/jOIP/+1bEAJas4ClOOq
                      X-Received: by 2002:adf:fac3:0:b0:2d9:5608:ee0 with SMTP id a3-20020adffac3000000b002d956080ee0mr975396wrs.69.1680606526736;
                      Tue, 04 Apr 2023 04:08:46 -0700 (PDT)
                      ARC-Seal: i=1; a=rsa-sha256; t=1680606526; cv=none;
                      d=google.com; s=arc-20160816;
                      b=x+DFD1IeUO5vuYcSgsgeHJqAsG3KUdotzRvQbYMS8sSmmajWXAqIGCz+Dj5nX2Uzm2
                      oquc2R6j2SimE5PWHbMgZ+3mloVmCvNImUeJF3zf2jEqiAW4oUwQFLdlO2vANdmefPqW
                      QxPn10yFfcQSWFepHO1Pm6RBtxqQxYuHfPKKvQP4FVTh6cs6Tq0O8/3AIssiy5/0Ver5
                      HZwwNhJ4bK++t8yybf5YMwkPT2HwbjSM67iYZlPTPX57Rry+MOnyw7pnYVR/YNeBz8/Z
                      eSM4lHPoOCGYmWfXreq3Ybl7cLIiDUzS5k8Temp+Eza9mLb+xwjplQI1R9rZRjUfnjMi
                      AWiQ==
                      ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
                      h=feedback-id:message-id:thread-topic:thread-index:references
                      :mime-version:date:to:from:subject:dkim-signature:dkim-signature;
                      bh=0bev6RJvUUQuD2OGslKi4yScwE5GFsQf7Y46VJj0DDo=;
                      b=VufS7Ck9XbxFiCd0y+UL6Xt9x3jbCW4Rb7YTNqZNKCWvDFZ59qtE14ANsun/LugOHx
                      E7AQRaF8vXY+WX7OP5fqXZAPtDzbl1srhZrNnYOTeiZTzdNtW0NbMrR+HOXOxtKlqUX9
                      K4y0mB3BSYG0a6gODA6pUnjdcyymVxCYuCnGVlTD4lz4szyzn36M+x6WQS90ik/gBuDw
                      jAyxIzyNpjavdWK0A3MCqWV7rzYw4KcLjGFGr3ZLAk+BG0E/sjyBzOnh7ho4tv1fafj8
                      F3cLl6HB0xQr2zYQuQCB5HRTIe4idjTStuYOAugIKarGDpl59Xd6wPQvo5SBhEhu2iHN
                      augA==
                      ARC-Authentication-Results: i=1; mx.google.com;
                      dkim=pass header.i=@esmero.eu header.s=vr4lk5ymt4znqsinu7p6tldo3p7npuxk header.b=DlPV4jLU;
                      dkim=pass header.i=@amazonses.com header.s=shh3fegwg5fppqsuzphvschd53n6ihuv header.b=eAZNBHqp;
                      spf=pass (google.com: domain of 010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com designates 54.240.2.78 as permitted sender) smtp.mailfrom=010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com;
                      dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=esmero.eu
                      Return-Path: <010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com>
                      Received: from a2-78.smtp-out.eu-west-1.amazonses.com (a2-78.smtp-out.eu-west-1.amazonses.com. [54.240.2.78])
                      by mx.google.com with ESMTPS id d3-20020adffd83000000b002c406802ab8si5090796wrr.596.2023.04.04.04.08.44
                      for <grufik.k@gmail.com>
                      (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
                      Tue, 04 Apr 2023 04:08:46 -0700 (PDT)
                      Received-SPF: pass (google.com: domain of 010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com designates 54.240.2.78 as permitted sender) client-ip=54.240.2.78;
                      Authentication-Results: mx.google.com;
                      dkim=pass header.i=@esmero.eu header.s=vr4lk5ymt4znqsinu7p6tldo3p7npuxk header.b=DlPV4jLU;
                      dkim=pass header.i=@amazonses.com header.s=shh3fegwg5fppqsuzphvschd53n6ihuv header.b=eAZNBHqp;
                      spf=pass (google.com: domain of 010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com designates 54.240.2.78 as permitted sender) smtp.mailfrom=010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com;
                      dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=esmero.eu
                      DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=vr4lk5ymt4znqsinu7p6tldo3p7npuxk; d=esmero.eu; t=1680606524; h=Subject:From:To:Date:Mime-Version:Content-Type:References:Message-Id; bh=0bev6RJvUUQuD2OGslKi4yScwE5GFsQf7Y46VJj0DDo=; b=DlPV4jLUViMtLGUyDxEXe4NWORA3OipMumm+ks88d9RxQ9Yt0/BTg7qCoQwCg9pH iRE6P/co9n4diiTeYJewlrftOsU489NdnhYh0ElrR0YKmJswYy8fmQVJM0ItyYIZhTf KTGl08EDxkBlsNk7HZ0RfLyi7W3LD21Pk2P+Hxcqf3z+2QTS9bvtmgBkpAvHhuB7G3x IekyPQR2gGPHWDKOpBXZA+1Fv7xg/UW3JBffS1hcueQJ/43HaqFKpCUNxw1hVtkoTrp yyaA5gCYKAp3KBZ8KoQfvOoUUZiY+/W6GdxSsroz3OlVR+Ss5s9D9NthIdMqaWKv49R HulyjwJswg==
                      DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=shh3fegwg5fppqsuzphvschd53n6ihuv; d=amazonses.com; t=1680606524; h=Subject:From:To:Date:Mime-Version:Content-Type:References:Message-Id:Feedback-ID; bh=0bev6RJvUUQuD2OGslKi4yScwE5GFsQf7Y46VJj0DDo=; b=eAZNBHqp/uTpayfapMU1AtLInInCk+0k99FwVRvUq4K8HgJEfYOlTSbI3znr3/uR j0felBp74gup8Ra0IcHxNnc5knn6pJLkKgMIDlkK41T2iB55wf1oVji1i8YSWMQkcKB OhrOgUr2zMfF5V+ILtK0h0G/4OAyNm+zlqeeda00=
                      Subject: Test Duplicate
                      From: Jakub Grufik <jakub.grufik@esmero.eu>
                      To: "grufik.k@gmail.com" <grufik.k@gmail.com>
                      Date: Tue, 4 Apr 2023 11:08:44 +0000
                      Mime-Version: 1.0
                      Content-Type: multipart/alternative;
                      boundary="=_Xk0bnbO22557egI6zcnflXvkP7lc5yO-3fj8SskPXEy+ZtEx"
                      References: <3875b49502182d7d470e02efc212a176/1680606520/8170@espo>
                      X-Mailer: Amazon WorkMail
                      Thread-Index: AQHZZuXQLMbunL+vRAacgFdTqWh5sQ==
                      Thread-Topic: Test Duplicate
                      X-Wm-Sent-Timestamp: 1680606523
                      Message-ID: <010201874bf47298-4ccaa75b-5f53-4ab4-bbbd-1f708239465e-000000@eu-west-1.amazonses.com>
                      Feedback-ID: 1.eu-west-1.b24dn6frgCi6dh20skzbuMRr7UL8M6Soir/3ogtEjHQ=:AmazonSES
                      X-SES-Outgoing: 2023.04.04-54.240.2.78
                      
                      --=_Xk0bnbO22557egI6zcnflXvkP7lc5yO-3fj8SskPXEy+ZtEx
                      Content-Type: text/plain; charset=UTF-8
                      Content-Transfer-Encoding: quoted-printable
                      
                      S pozdravem,
                      
                      Jakub Gruf=C3=ADk
                      V=C3=BDvoj=C3=A1=C5=99 CRM
                      
                      tel.:=C2=A0+420=C2=A0725 146 100
                      web:=C2=A0www.esmerocz.eu
                      mail.:=C2=A0jakub.grufik@esmerocz.cz
                      fb.:=C2=A0esmerozimnizahrady
                      instagram:=C2=A0esmerocz
                      youtube:=C2=A0esmero
                      
                      
                      --=_Xk0bnbO22557egI6zcnflXvkP7lc5yO-3fj8SskPXEy+ZtEx
                      Content-Type: text/html; charset=utf-8
                      Content-Transfer-Encoding: base64​
                      The original message ID is stored in the attribute "References"
                      Attached Files

                      Comment


                      • #14
                        Not good. This would be tricky to distinguish from a legit reply email. As the references header is supposed to contain the ID of the email the current email is replied to.

                        Comment


                        • Jakub Grufik
                          Jakub Grufik commented
                          Editing a comment
                          can it be somewhere in the settings of the project that every email is being archived automaticaly? I tried to look for it but without success.. for me it would make sense that problem with duplicates would be if both emails have status "sent" for example..but one is sent the other one is archived, so I do not know why the system is archiving every single email..

                      • #15
                        Hello guys, can someone help me to solve this issue, please? We have a looot of unnecessary emails in our system and basically, everything is being duplicated with status "Archived".

                        I dont think this is caused by the fact that amazon is changing the ID of the message. In that case it would duplicate everything with status "Sent" or not? One email has status "Sent" and the duplicate has status "Archived". Why is this happening please

                        Our system is srsly overloaded with emails atm.

                        Comment

                        Working...
                        X