Proposal: Record archiving / data retention (willing to implement)

[Makuro]

Hi,

on long-running instances the email, note and attachment tables keep growing, which slows down list views, streams and search and inflates backups. The existing “Archive” folder for emails only moves records, it doesn’t reduce data. The forum has many threads about this; the usual answers are manual SQL deletes or custom cleanup jobs.

I’d like to build a proper archiving feature and contribute it. Rough idea:

• Archive storage: a single generic table (e.g. ArchiveRecord) holding a JSON snapshot of the record including resolved relations, a denormalized text field with a FULLTEXT index, the parent (Account/Contact) and a snapshot of team/assigned-user access.
• Archiving job: a scheduled job with configurable rules per entity type (e.g. “Case closed > 2 years”, “Email older than X and not linked to an open record”). It snapshots, moves attachments (optionally to a separate storage), then hard-deletes the original together with related stream notes. It supports a dry-run mode.
• Read-only UI: ArchiveRecord as a regular entity with read-only ACL, a detail view that renders the snapshot, and an “Archive” panel on Account/Contact.
• Retention and GDPR: a retention period per type after which archived records are purged, and a way to remove all archived data related to a person.
• Restore: restoring a single record from its snapshot, if feasible.

Questions to the maintainers:

1. Would you consider this for core, or should it be an extension?
2. If extension: would you accept small core PRs that make it cleaner, e.g. a reliable way to collect all relations of a record before deletion?
3. Any architectural constraints or existing plans I should align with?

I’m happy to adapt to your conventions.

Thanks!

[emillod]

I can't say on behalf of EspoCRM, i'm not connected to them in any way. If i could guess, i would say:
- If you would propose a solid solution they would probably consider it, but you should start with extension for sure
- now a lot of people trying to accomplish things via AI, so probably devs will not be optimistic to new things without checking before and they already have a lot on their shoulders
- you can send a PR on github, but you have to accept their licensing

[eymen-elkum]

This would be useful for many long-running EspoCRM installations. I think starting with an extension is the most practical approach: it lets people use and test the feature without waiting for it to become part of core.

The EspoCRM team has its own roadmap and priorities. Archiving may already be under consideration, but we cannot assume whether or when it will be implemented.

At Eblasoft, we’ve built several features that benefited many users, with similar functionality later becoming part of EspoCRM core. Some of our solutions were in use for more than five years before that happened.

A few safeguards would be worth defining before implementation, especially if using AI-assisted development:

- Follow EspoCRM’s documented extension mechanisms and the target version’s APIs. Avoid core modifications, and ask the AI to verify existing APIs rather than invent methods or metadata options.
- Treat permissions as a backend requirement. A snapshot of old teams or ownership should not preserve access that a user has since lost. Archived records, search results and attachments all need access checks.
- Define how shared attachments, relationships and stream records will be handled before deleting anything. An attachment linked elsewhere must not disappear with an archived record.
- Start with dry-run and copy-only modes. Do not delete originals until the archive has been verified and recovery has been tested.
- Make scheduled jobs resumable and safe to retry, with small batches, clear logs and protection against overlapping runs.
- Define exactly what “restore” supports, including missing related records, changed fields and schema upgrades. A JSON snapshot alone is not proof that a record can be restored correctly.
- Test with fictional data on a separate instance, including permission changes, interrupted jobs and failed restores. Do not share production records or credentials with AI tools.

I’d give these constraints to the AI upfront and ask it to flag uncertainties rather than silently make assumptions. Starting with one entity type and a small, well-tested scope would make the first release much easier to review and trust.

[yuri]

Hi,

I'm bit skeptical about such a feature being a good fit for a general purpose CRM.

Another problem, is that it would be quite hard to design it right. Storing a snapshot with all relationships and restoring it back can have many edge cases.