Inbound email silently lost when Message-ID exceeds 255 chars

[berlinerkind]

Version: 10.0.3 (entityDefs unchanged on current master: messageId maxLength 255, messageIdInternal 300)

Steps to reproduce:
1. Personal or group account fetches a message whose Message-ID header is longer than 255 chars (RFC 5322 sets no limit).
2. Import fails: ERROR: (22001) Import message error. EmailAccount <id>. PDOException: SQLSTATE[22001]: ... 1406 Data too long for column 'message_id'

Observed:

• No email record is created.
• UID is still marked processed → the message is never retried.
• No admin-visible signal beyond the log line. Silent permanent loss.

Real-world senders that trigger it:

• Microsoft 365 notification mails (…@odspnotify: Teams missed activity, guest invites, file shares) — 262–337 chars. Every M365 tenant emits these.
• Legacy Outlook <!&!AAA…base64…> IDs — 338 chars observed.
• Bulk mailers with tracking payloads in the ID — up to 424 chars observed.

On our instance: 20 failed import events in 4 weeks across 5 accounts; 25 messages lost, confirmed via IMAP header scan.

Expected: import must not fail on a standards-compliant header.

Suggested fix: widen messageId/messageIdInternal, plus defensive truncation (or hashing of overlong IDs) in DefaultImporter: processMessageId, width alone is not sufficient.

Workaround: custom entityDefs override "messageId": {"maxLength": 512}, "messageIdInternal": {"maxLength": 640} + rebuild. Index stays under the 3072-byte InnoDB limit (utf8mb4, DYNAMIC).

[yuri]

I believe truncation should not be used when storing IDs.

Setting the max length to 500 should be sufficient for most cases, yet it won't be 100% compliant. Not sure where it will be an easy upgrade for everyone if we ship the change in a hotfix release.

[yuri]

We might need to change the collation to ASCII and then increase the width to 998 (the max length defined by the standard). But this should not be done in a hotfix release cycle.

[berlinerkind]

Agree on 998 and ASCII.

Two points:

1. Width alone can't close the failure mode. Non-compliant senders exist (424 chars already observed; >998 or 8-bit garbage possible). The invariant needed: an unstorable Message-ID must never abort the import: on abort the UID is marked processed and the message is permanently lost, silently. That's the bug.
2. Overflow handling: for this reason hash it, don't truncate. Truncation risks false duplicate matches. sha256 for anything over the column width keeps dedup deterministic. Original header remains in stored message source.

Suggestion for the release problem:

- Hotfix: importer guard only (processMessageId: length check -> hash -> import proceeds). No schema change, upgrade-safe, stops the data loss now.
- Minor release: ASCII/998 schema change + messageIdInternal headroom, where a long ALTER is acceptable.

[yuri]

I don't think we need to incorporate hashing for now. It complicates things, produces data entries with no real ID, then in future, when we increase length, we will have potential duplicates. It's not that clean solution. Another problem, the 'messageId' field is used as a source of truth in the system, not just for duplicate checks.

500 characters (changed in hotfix) should be suffice for the real-world I believe. I doubt there will be any email losses after it.

> That's the bug.

I don't agree with this. Skipping items while logging the error is a deliberate choice. Another question is to make sure we fully support valid messages and they are never skipped. Worth noting that the skipped message is not lost 'permanently', it's still on the IMAP server. And I'd avoid "it's a bug" argumentation – it may come off harsh.

I'd rather call it a day after the max length 500 change for now. Currently, there are lots of other things we have to focus on. It's really much more complex issue to solve it fully (if we opted to support invalid message IDs) than it may seem, if we consider all the factors such as not breaking contracts, migrating existing instances (with millions of emails) we don't have direct control over.

While, for PostgreSQL users it's simple to fix: just increase the max length to 1000.