Upload all files to an object store

[yuri]

If files will be stored in a single directory w/o subdirectories it will be easy to migrate. You will need to carry over all files from data/upload to your storage, run script in mysql that updates storage column of record in attachment table. And then you will need to write the implementation class for fetching, pushing, removing.

But first, I need to finish all the work.

[wtconseil]

Whoww ! great !

how do you plan to help people to migrate to this kind of Storage Backend ?
Meaning... I have a setup in production for more than 2 years now and many files.

Should i migrate all the data/upload content manually to storage backend or could we have a small script to run that would use these controllers etc.. ?

Thanks for your support and your good job !!

[yuri]

Here the commit: https://github.com/espocrm/espocrm/c...f61f06ff78d64d
Not finished but the most important part is implemented

[wtconseil]

Yes sure! But I have that this upload directory could act as a temp directory.
If we lost it... it doesn't matter.

All "expected" files will be in the object store.

[yuri]

I think upload directory will be used anyway for storing files temporary when they are needed to be accessed by some existing implementations like Email Sending, Image transformation. It's easier for as because doesn't require us to rewrite a lot of code.

[yuri]

Yes. We have it on our roadmap. We will create a framework. But you will need to implement a wrapper class.

[huscmk]

Hi,

Still no idea when it could be integrated ?

Thanks

[wtconseil]

I currently use a "technical solution" called Cloudfuse ( https://github.com/redbo/cloudfuse )
it mounts a "swift container" (like a s3 bucket / cloud folder) and sync the file over http apicall

But it don't think it could be interesting to continue with that over the time...

Let me know if you have some plan to develop that "cloud folder" connector in a close futur?
It's not possible for me any more to have data/upload folder directly on the filesystem any more :-(

[wtconseil]

Hi Yuri

Let me know if you have some plan about that soon ?
Thanks

[yuri]

Hi,

It was planned but no ETA available.

[wtconseil]

Hi

i'm thinking about deploying EspoCRM on a docker container to have an immutable image.. and orchestrate my deployment through Kubernetes.

EspoCRM will be running in HA mode, on many nodes, behind a load-balancer.

I have currently a design issue regarding the upload folder.

Is there a way (or is it in the roadmap) to have a connector to upload content to an "object-store" backend (ex: AWS S3 or Openstack Swift for example) rather than putting files in the local machine?

Currently, because of this design, i have to sync between containers the upload folder.. (or bind-mount a volume..) between the host and the container..
I think that it could be great to have this feature.

SugarCRM (that was used before my POC) seems to have this uploadStream class:

https://github.com/saeed237/sugarpro7.1/blob/master/include/SugarUploadS3.php

and we can configure the upload class wrapper in config file.

That could be really awesome i guess to have that for Espo

Thanks!