Announcement

Collapse
No announcement yet.

Upload all files to an object store

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • wtconseil
    started a topic Upload all files to an object store

    Upload all files to an object store

    Hi

    i'm thinking about deploying EspoCRM on a docker container to have an immutable image.. and orchestrate my deployment through Kubernetes.

    EspoCRM will be running in HA mode, on many nodes, behind a load-balancer.

    I have currently a design issue regarding the upload folder.

    Is there a way (or is it in the roadmap) to have a connector to upload content to an "object-store" backend (ex: AWS S3 or Openstack Swift for example) rather than putting files in the local machine?

    Currently, because of this design, i have to sync between containers the upload folder.. (or bind-mount a volume..) between the host and the container..
    I think that it could be great to have this feature.

    SugarCRM (that was used before my POC) seems to have this uploadStream class:
    https://github.com/saeed237/sugarpro...arUploadS3.php

    and we can configure the upload class wrapper in config file.

    That could be really awesome i guess to have that for Espo

    Thanks!


  • wtconseil
    replied
    Originally posted by dabasystem View Post
    In the company that we work, we have developed it. You can contact us by email, I guess there will be no problem!
    cau@dabasystem.com
    Hi
    can you share the code?

    Leave a comment:


  • dabasystem
    replied
    In the company that we work, we have developed it. You can contact us by email, I guess there will be no problem!
    cau@dabasystem.com

    Leave a comment:


  • bandtank
    replied
    Originally posted by dabasystem View Post
    HI!

    I have a good news and another bad news ...
    The good new:
    I have managed to put the s3 and it works perfectly: D: D

    The bad new:
    It only works for me in EspoCrm 4.7 I've tried to do the same configuration in 5.5, 5.4 and I can't get it to work.

    Is there a change in the FileStorage? What I can do?

    When has it functional I will upload the files in case I can help someone
    Thank you so much!
    This would be absolutely amazing if we can get it to work in 5.5+. Storing files on S3 would completely change my backup and recovery strategy for the better.

    Leave a comment:


  • dabasystem
    replied
    HI!

    I have a good news and another bad news ...
    The good new:
    I have managed to put the s3 and it works perfectly: D: D

    The bad new:
    It only works for me in EspoCrm 4.7 I've tried to do the same configuration in 5.5, 5.4 and I can't get it to work.

    Is there a change in the FileStorage? What I can do?

    When has it functional I will upload the files in case I can help someone
    Thank you so much!

    Leave a comment:


  • bandtank
    replied
    I'm very curious about this as well. Being able to upload and download files to S3 would be a huge upgrade to local file storage. It would truly make EspoCRM highly available since the front end could be clustered. This would also be a necessity for Docker containers.

    Leave a comment:


  • huscmk
    replied
    Hi, Did anyone completed the AWS S3 implementation by any chance ? Regards,

    Leave a comment:


  • yurikuzn
    replied
    1. In data/config.php need to add parameter 'defaultFileStorage' => 'AmazonS3'


    2. Create custom/Espo/Custom/Resources/metadata/app/fileStorage.json with the following content
    PHP Code:
    {
        
    "implementationClassNameMap": {
            
    "AmazonS3""\\Espo\\Custom\\Core\\FileStorage\\Storages\\AmazonS3"
        
    }


    3. Create file custom/Espo/Custom/Core/FileStorage/Storages/AmazonS3.php

    PHP Code:
    <?php

    namespace Espo\Custom\Core\FileStorage\Storages;

    use \
    Espo\Entities\Attachment;

    use \
    Espo\Core\Exceptions\Error;

    class 
    AmazonS3 extends \Espo\Core\FileStorage\Storages\Base
    {
        protected 
    $dependencyList = ['fileManager''config'];

        protected function 
    getFileManager()
        {
            return 
    $this->getInjection('fileManager');
        }

        protected function 
    getConfig()
        {
            
    // in config we can store storage credentials
            
    return $this->getInjection('config');
        }

        public function 
    unlink(Attachment $attachment)
        {
            
    // removing file, returns true or false
        
    }

        public function 
    isFile(Attachment $attachment)
        {
            
    // checks whether file exists
        
    }

        public function 
    getContents(Attachment $attachment)
        {
            
    // returns contents of the file
        
    }

        public function 
    putContents(Attachment $attachment$contents)
        {
            
    // stores contents of the file
        
    }

        public function 
    getLocalFilePath(Attachment $attachment)
        {
            
    // returns local file path of copied temporary file to be able to download//
            // if user wants to download the file we need to have it copied on the server as a temporary file
            // maybe this can be improved in future
        
    }

        public function 
    getDownloadUrl(Attachment $attachment)
        {
            
    // if there is a direct url to download; likely, leave this method empty
        
    }

        public function 
    hasDownloadUrl(Attachment $attachment)
        {
            
    // whether there is a url for download; likely return false;
        
    }
    }
    Don't forget to clear cache after changes in metadata.

    You can use $attachment->id for filenames on the storage

    I'm very busy these days. Don't have time to try to make this implementation.
    Last edited by yurikuzn; 03-14-2017, 03:37 PM.

    Leave a comment:


  • huscmk
    commented on 's reply
    That would indeed be an excellent plus-value if we could externalize media files to Amazon S3 as it would mean no more storage issues on the servers running espocrm.

    As I understand, there is now a framework with a storage manager that still needs to get a wrapper for S3 (or any other object storage with an API) but it seems hard to do by the look to that code only as it needs a quite good understanding of how media management is done in espocrm.

    Anyone who could help with the task ?

  • bandtank
    replied
    yurikuzn,

    Could you please explain how to use the file storage manager feature? I would like to upload files to S3 or Dropbox, but I don't understand if that's possible or how to get started.

    Leave a comment:


  • bandtank
    replied
    I see the release is live and the upload feature is listed in the Github release notes. Thanks for your hard work.

    Leave a comment:


  • wtconseil
    replied
    Awesome :-))))
    Thanks for your hard work!

    Leave a comment:


  • yurikuzn
    replied
    In 4.5.0 release there will be a framework providing an ability to implement external storage. Release is planned for today.

    Leave a comment:


  • bandtank
    replied
    Is there anything the community can do to help? I really want this feature and I am willing to contribute if there's a way to speed up the release. I want to start using EspoCRM, but storing files in a local upload folder is a deal breaker because the ephemeral nature of an EC2 instance is very risky.

    Leave a comment:


  • yurikuzn
    commented on 's reply
    In the next 4.5.0 release. Soon.
Working...
X