Proposal: Record archiving / data retention (willing to implement)

[Makuro]

Hi,

on long-running instances the email, note and attachment tables keep growing, which slows down list views, streams and search and inflates backups. The existing “Archive” folder for emails only moves records, it doesn’t reduce data. The forum has many threads about this; the usual answers are manual SQL deletes or custom cleanup jobs.

I’d like to build a proper archiving feature and contribute it. Rough idea:

• Archive storage: a single generic table (e.g. ArchiveRecord) holding a JSON snapshot of the record including resolved relations, a denormalized text field with a FULLTEXT index, the parent (Account/Contact) and a snapshot of team/assigned-user access.
• Archiving job: a scheduled job with configurable rules per entity type (e.g. “Case closed > 2 years”, “Email older than X and not linked to an open record”). It snapshots, moves attachments (optionally to a separate storage), then hard-deletes the original together with related stream notes. It supports a dry-run mode.
• Read-only UI: ArchiveRecord as a regular entity with read-only ACL, a detail view that renders the snapshot, and an “Archive” panel on Account/Contact.
• Retention and GDPR: a retention period per type after which archived records are purged, and a way to remove all archived data related to a person.
• Restore: restoring a single record from its snapshot, if feasible.

Questions to the maintainers:

1. Would you consider this for core, or should it be an extension?
2. If extension: would you accept small core PRs that make it cleaner, e.g. a reliable way to collect all relations of a record before deletion?
3. Any architectural constraints or existing plans I should align with?

I’m happy to adapt to your conventions.

Thanks!

[emillod]

I can't say on behalf of EspoCRM, i'm not connected to them in any way. If i could guess, i would say:
- If you would propose a solid solution they would probably consider it, but you should start with extension for sure
- now a lot of people trying to accomplish things via AI, so probably devs will not be optimistic to new things without checking before and they already have a lot on their shoulders
- you can send a PR on github, but you have to accept their licensing

[eymen-elkum]

This would be useful for many long-running EspoCRM installations. I think starting with an extension is the most practical approach: it lets people use and test the feature without waiting for it to become part of core.

The EspoCRM team has its own roadmap and priorities. Archiving may already be under consideration, but we cannot assume whether or when it will be implemented.

At Eblasoft, we’ve built several features that benefited many users, with similar functionality later becoming part of EspoCRM core. Some of our solutions were in use for more than five years before that happened.

A few safeguards would be worth defining before implementation, especially if using AI-assisted development:

- Follow EspoCRM’s documented extension mechanisms and the target version’s APIs. Avoid core modifications, and ask the AI to verify existing APIs rather than invent methods or metadata options.
- Treat permissions as a backend requirement. A snapshot of old teams or ownership should not preserve access that a user has since lost. Archived records, search results and attachments all need access checks.
- Define how shared attachments, relationships and stream records will be handled before deleting anything. An attachment linked elsewhere must not disappear with an archived record.
- Start with dry-run and copy-only modes. Do not delete originals until the archive has been verified and recovery has been tested.
- Make scheduled jobs resumable and safe to retry, with small batches, clear logs and protection against overlapping runs.
- Define exactly what “restore” supports, including missing related records, changed fields and schema upgrades. A JSON snapshot alone is not proof that a record can be restored correctly.
- Test with fictional data on a separate instance, including permission changes, interrupted jobs and failed restores. Do not share production records or credentials with AI tools.

I’d give these constraints to the AI upfront and ask it to flag uncertainties rather than silently make assumptions. Starting with one entity type and a small, well-tested scope would make the first release much easier to review and trust.

[yuri]

Hi,

I'm bit skeptical about such a feature being a good fit for a general purpose CRM.

Another problem, is that it would be quite hard to design it right. Storing a snapshot with all relationships and restoring it back can have many edge cases.

[Makuro]

Thanks a lot for the feedback, all of you.

yuri – fair point, and I share the concern. We run a long-running instance ourselves, and the dependencies between emails, notes, attachments and their parents are exactly what makes this tricky. A snapshot that claims to be restorable but silently loses relations would be worse than no archive at all.

At the same time, on instances with years of mail traffic this is simply something daily operations needs. Today the options are “keep everything forever” or “delete with SQL and hope nothing breaks”, and neither is great.

So I’ll take the advice here and go much narrower:

• Extension, not core. No core modifications.
• One entity type first: Email, since it is by far the largest table on long-running instances.
• No restore. Archive is one-way: read-only view and export only. That avoids most of the edge cases you mention.
• Access is checked against current permissions (via the parent record), not a snapshot of old teams. Thanks eymen-elkum, good point.
• Shared attachments are never removed while they are still referenced elsewhere.
• Dry-run and copy-only modes first. Originals are deleted only as a separate, explicit step after verification. Jobs run in small, resumable batches.

I’ve set up a separate test instance with generated fictional data (100k+ emails, cases and stream notes) to measure the impact before and after.

yuri, one question so I don’t build against the grain: when an email is deleted, is there a recommended way for an extension to find everything that references it (notes, attachments, relation tables), or should the extension handle each of these explicitly?

I’ll post back here once there is something to try.

[yuri]

> when an email is deleted, is there a recommended way for an extension to find everything that references it (notes, attachments, relation tables)

There’s no built-in implementation for this. Code would need to check all relationships. While for emails it can be assumed as a fixed list of relationships, for regular entity types one would need to loop through all relationships and check their entity types.