Incorrect processing of bounced emails

[MatLudlam]

Hi, I have set up a campaign which is email based. This sends out the emails, and the bounced emails come back to a given email box, where they are read by Espo.

Espo then processes the bounced emails, and updates the campaign statistics accordingly. I can then see the bounced emails by applying the filters within the campaign. So far so good, all works as expected.

Now the issue. Not all of the bounced emails are processed correctly by Espo.

I have attached one that is not correctly recognised (and I have plenty more).

I have analysed the email with a tool I put together, and you can see the output as a second attachment.

You will notice that the email contains a part with a Content Type of "message/delivery-status". I have not dumped the headers (but happy to if needed), however one of those headers is a "Finial-recipient" one, and it specifies "cristina.tudorache@star-storage.ro" who was the person for whom the email was originally destined, and has left.

So, if processed correctly cristina.tudorache@star-storage.ro should be tagged as bounced, however it is not.

Can you let me know why please?

PS - I would not normally share email addresses on a forum, but they are invalid so I thought it ok.

[yuri]

I think bounced are process properly. But some mail providers do not bother to send a proper bounced messages. Can we cover them all? No. Can we try to achieve coverage of the most popular providers? Maybe. But not always. It can be problematic.

Try enabling VERP (Administration > Outbound Emails > Use VERP). This can improve bounce handling, but your email provider should support it.

Having only email address is not enough for bounced message. It should contain a unique ID that we send with the mass email message.

[MatLudlam]

Hi Yuri - the bounced email is from Office 356 (who are our provider), so I would say a provider that matters.

The email that was returned is correct, it has the part with Content type: message/delivery-status, and that part has the correct header and the correct email.

I had 300+ emails bounce in 1 campaign. Only about 150 of them were processed correctly. I don't know what ESPO is going. What I do know, is that of the 300 mails about 295 of them all have the message/delivery-status and the correct header

I would also say that a bounce should update the initial Contact / Lead record to say that the address is invalid, not just record it within the campaign. There is no point in sending any more emails to them.

I would also say that the last part of the email contains the original email that was sent, so any token embedded within that can be extracted.

I have also turned on VERP so we will see if it is any better as there is another mail shot going out tomorrow.

[yuri]

The bounced email should contain a unique ID. We extract it to know that the email is authentic. Otherwise one can forge bounced emails and do a harm to the business. VERP makes the provider to include the ID as a part of the "TO" address.

The approach to try to extract something in different ways depending on a provider is not good. Companies like Mircrosoft like to change response formatting often. We will stick to rely on VERP. I believe MS should support it.

Could you attach the full email original that I can open in a text editor?

[yuri]

> I would also say that a bounce should update the initial Contact / Lead record to say that the address is invalid, not just record it within the campaign. There is no point in sending any more emails to them.

Not all bounced mean that email is invalid. There are hard and soft bounced types. Soft bounced should not be marked as invalid. Example of soft bounced: the user exceeded the disk space limit.

[MatLudlam]

> I would also say that a bounce should update the initial Contact / Lead record to say that the address is invalid, not just record it within the campaign. There is no point in sending any more emails to them.

Not all bounced mean that email is invalid. There are hard and soft bounced types. Soft bounced should not be marked as invalid. Example of soft bounced: the user exceeded the disk space limit.

Whilst I agree you are technically correct, of the 300+ email bounced that occurred, none of them were because of low disk space. So now I have 300 bounced items in a campaign, and I need to find some way of updating the Contact records manually to reflect that they have bounced. If I keep sending emails to bounced addresses then eventually I will get black-listed. The easy way of doing this is to just mark the Contacts as invalid. What else would you suggest?

Separately I have seen a request for the sample email again (cannot see it here). How would you like me to sent it to you. When I move from Outlook to the desktop it does something to the format of the mail. I can try and send to you as an attachment and also try to just forward it to you. Please let me know.

[yuri]

There should be something like "Show Original" in the email client.

[MatLudlam]

There should be something like "Show Original" in the email client.

Interestingly Outlook has a different set of menus when it has a delivery failure, and that is not one of the options.

I did try "Forward as attachment" to my Hotmail address and it came through correctly. I have also just tried a standard "Forward" but that destroys the information within it.

I have created a PST with 6 sample emails, all of which are processed incorrectly. The only one that I have investigated fully is cristina.tudorache@star-storage.ro. I can confirm that this has the bounced header with correct email address. It also has the original email that was sent within in. In Outlook you can even press "Send Again" and it will pop up the original email. All the mails are in the "Samples" folder.

As the PST is too big to upload here, you can get it from here: https://1drv.ms/u/s!AiQ6JPOdP3xck7VCSmfcnfug9RPrzg?e=p47jvK

[yuri]

Unfortunately, I don't have Outlook to open PST files.

[MatLudlam]

Unfortunately, I don't have Outlook to open PST files.

Is there an email account I can forward it to as an attachment?

[yuri]

Maybe try to upload on the customer portal? As an attachment of a case.

[MatLudlam]

The question is, what do I upload? If I take the mail out of Outlook, it destroys the data within it (that is what I did originally). Apart from a PST, the only other way that I can send it, without destroying the content, is to forward as an attachment.

Actually try the ZIP version of an MSG file which is on that share above because it is too big to attach. I am not sure how you will open the MSG, but the attachment (which is what you are interested in) is ok.

As a different idea, this is in our Cloud Instance of ESPO, so you can log into that and look at it if you want.

[espcrm]

I'm guessing it need to be some plaintext format for Yuri to be able to open it, something that can be open with Notepad.. I think .msg is the standard format that can do just that. PST or EML is outlook locked?

Don't quote me on this though, I tried searching the difference in format but internet slow and I want to read next thread...

EDIT: Here you go: https://www.shoviv.com/blog/how-to-s...-and-msg-file/
EML is the plain text format.

[MatLudlam]

I have put both a PST and MSG format files on this share: https://1drv.ms/f/s!AiQ6JPOdP3xck7VCSmfcnfug9RPrzg

The issue is that messages have multiple parts embedded within each other. So when viewed in Outlook that email just presents an error message in an HTML page. Everything is there, but as the HTML is built for someone to read, it is not easy to process. In the email are another 3 (actually more if you include the wrappers) parts that are:
* A plain text version of the error message
* The original HTML message
* A "message/delivery-status" which has a very specific format, and it machine readable.

Outlook only shows the 1st of those parts (the error message) but the other 3 parts are there. When creating a MSG file, Outlook simplifies things and removes some of the other parts.

I have tools that I am happy to show working that I have put together to illustrate this. They are written in Java (my preferred language) rather than PHP.

[yuri]

I wrote a test that checks bounced emails recognition. Any not recognized bounced email can be added as a test-case and it will be easy to fix. All is needed is a raw contents of such emails.

https://github.com/espocrm/espocrm/blob/fix/tests/unit/Espo/Core/Mail/Account/BouncedRecognizerTest.php

[MatLudlam]

Yuri - thanks for this. Do you have what you need on the above share? If not, what can I provide you?

Separately, where is the source for BouncedRecognizer? I have tried looking through the Git repository, but cannot find it (not very good at Git!!). Thanks

[MatLudlam]

Hi yuri I am still having issues with this. My questions above still stand:

• How do you want me to share the emails that are not correctly processed with you?
• Where can I see the source for BouncedRecognizer?

One thing I could do is create a new Hotmail account, put the bounced emails in it, then share the details with you so that you can connect directly to the mailbox from your ESPO instance and see what is happening. I would not want to post the access details here though.

Thoughts?

[MatLudlam]

Hi Taras and yuri can I get your feedback please. This is incorrect functionality; I have provided what is needed for you to re-product it.

It would be great to get some feedback one way or another.

Thanks.

[yuri]

> I have provided what is needed for you to re-product it.

Sorry but it's not true. I didn't get an email original from you. I made some fixes that potentially could fix the problem. These fixes are in v7.1, not sure if your instance is upgraded.

[MatLudlam]

Hi yuri - in post 14.3 above are steps to re-produce the problem along with sample email addresses.  Can you test these on the new version?

You also do not say anywhere above that you have fixed it. I have asked many times for feedback, but it was never given.

You are correct, I am not on 7.1 yet, happy to have that conversation.  We are on the ESPO Cloud, so if your collages there want to contact me, happy to do it.