Group Email: Only some emails are fetched from IMAP inbox

[rubensolvang]

When connecting to my IMAP email account, only some of the emails in the inbox are fetched, at random. Both the email account and the EspoCRM installation is quite new and 'clean'.

I have tried removing and adding the IMAP account several times, playing with different settings, marking all emails in the email account as unread etc. Anyone else that have experienced this problem?

My install is running on a shared host (HawkHost), with all requirements met. The cronjob is working. Are there any hidden filters / rules that could interfere with what emails are fetched?

[rubensolvang]

I tried using a different email account, which seems to work fine. Has to be a problem with the email accounts hosted on my specific host, HawkHost – none of them worked as expected. I made a free account at gmx.com, forwarded my emails to this address and used the gmx.com to fetch via IMAP.

[tobias]

I experience the same issue, but won't be able to relay via a third party account for security reasons.

Any other ideas to fix this?

---

EDIT 1:

Field fetch_data shows the correct number of emails:

{"lastUID":{"INBOX":"4"},"lastDate":{"INBOX":"2016-08-10 08:25:53"}}

However, these for emails wouldn't show up under "EMails" and Cases won't be created either.

[yuri]

Could emails be placed in different folders on IMAP server?

[tobias]

It's a brand new mail account with only 4 test emails in the root folder. All folders seem to be detected by Espo just fine (see screenshot).

We've selected "INBOX" but the retrieved emails haven't been synced (fetch date was set way in the past).

It appears, that only SOME e-mails are being ignored, as already mentioned by the OP.

And since we haven't set any filters ourselves, do you apply any filters when fetching emails?

[yuri]

Can you attach or send raw data of any not imported emails? We will investigate.

There shouldn't be any filters by default.

[yuri]

Just checked email you provided and it was imported well.

[yuri]

Any related error in log? Maybe connection getting failed during import process?

data/logs

[tobias]

This could help:
[2016-08-10 07:58:03] Espo.ERROR: InboundEmail 57aadae2e63c98105 (Get Message): [0] Header value must be composed of printable US-ASCII characters and valid folding sequences. [] []

[yuri]

Does it happen with every not imported email? It could be a cause of the problem. This exception is thrown by zend mail library when getting message from imap.

[tobias]

Yes, we we can reproduce this error with emails that contain German language chars (English seems to be just fine).

[yuri]

* Are those emails sent by the same email client?

* The same emails were imported w/o problems from another IMAP provider?

[tobias]

Same client: these are script generated emails. We're using the Laravel framework's inherent mailing functionality (https://laravel.com/docs/5.2/mail) to send welcome emails to new users and a copy to our own mailbox.

Another provider: since these e-mails had been generated by the system, this could not be tested. However, we experience the same problem with emails received from various other providers (sent to the same IMAP mailbox and not being imported by Espo).

[yuri]

It could be not properly formed email header. Some clients accept, some (like zend mail) doesn't.

[tobias]

The library we use is called "SwiftMailer" and it claims that Headers are encoded properly in case non-ascii chars are included:

If the header contains any characters that are outside of the US-ASCII range however, they will be encoded, just like they are for text headers. This is nothing to be concerned about since mail clients will decode them back. Likewise, if the parameters contain any non-ascii characters they will be encoded so that they can be transmitted safely.

Source: http://swiftmailer.org/docs/headers....erized-headers

We've just checked and emails seem to be imported just fine if no non-ascii chars are used in the subject (itself being a part of the Header).

That said, there's nothing we could do on our end other than avoiding such chars in our system generated emails, which cannot be assured for emails coming from our users.

There seems to be a fix on GitHub: https://github.com/zendframework/zend-mail/issues/33

Could you please check if that would help?