Announcement

Collapse
No announcement yet.

Upload all files to an object store

Collapse
X
Collapse
 
  • Page of 3
  • Filter
  • Time
  • Show
Clear All
new posts
Previous 1 2 3 template Next
  • #1

    Upload all files to an object store

    Hi

    i'm thinking about deploying EspoCRM on a docker container to have an immutable image.. and orchestrate my deployment through Kubernetes.

    EspoCRM will be running in HA mode, on many nodes, behind a load-balancer.

    I have currently a design issue regarding the upload folder.

    Is there a way (or is it in the roadmap) to have a connector to upload content to an "object-store" backend (ex: AWS S3 or Openstack Swift for example) rather than putting files in the local machine?

    Currently, because of this design, i have to sync between containers the upload folder.. (or bind-mount a volume..) between the host and the container..
    I think that it could be great to have this feature.

    SugarCRM (that was used before my POC) seems to have this uploadStream class:
    sugarpro7.1/include/SugarUploadS3.php at master · saeed237/sugarpro7.1
    https://github.com/saeed237/sugarpro7.1/blob/master/include/SugarUploadS3.php
    Contribute to saeed237/sugarpro7.1 development by creating an account on GitHub.


    and we can configure the upload class wrapper in config file.

    That could be really awesome i guess to have that for Espo

    Thanks!

    Tags: None
    • Likes 1
  • #2
    Hi,

    It was planned but no ETA available.

    Comment

    • huscmk
      #2.1
      huscmk commented
      02-07-2017, 11:45 PM
      Editing a comment
      Hi,

      Still no idea when it could be integrated ?

      Thanks
  • #3
    Hi Yuri

    Let me know if you have some plan about that soon ?
    Thanks

    Comment

    • #4
      I currently use a "technical solution" called Cloudfuse ( https://github.com/redbo/cloudfuse )
      it mounts a "swift container" (like a s3 bucket / cloud folder) and sync the file over http apicall

      But it don't think it could be interesting to continue with that over the time...

      Let me know if you have some plan to develop that "cloud folder" connector in a close futur?
      It's not possible for me any more to have data/upload folder directly on the filesystem any more :-(

      Comment

      • #5
        Yes. We have it on our roadmap. We will create a framework. But you will need to implement a wrapper class.
        Last edited by yuri; 02-08-2017, 01:33 PM.

        Comment

        • #6
          I think upload directory will be used anyway for storing files temporary when they are needed to be accessed by some existing implementations like Email Sending, Image transformation. It's easier for as because doesn't require us to rewrite a lot of code.
          Last edited by yuri; 02-08-2017, 03:37 PM.

          Comment

          • #7
            Yes sure! But I have that this upload directory could act as a temp directory.
            If we lost it... it doesn't matter.

            All "expected" files will be in the object store.

            Comment

            • #8
              Here the commit: https://github.com/espocrm/espocrm/c...f61f06ff78d64d
              Not finished but the most important part is implemented

              Comment

              • huscmk
                #8.1
                huscmk commented
                02-20-2017, 09:51 PM
                Editing a comment
                Great news ! Will you tell us when it's finished ?

                Thanks !
              • yuri
                #8.2
                yuri commented
                02-21-2017, 11:00 AM
                Editing a comment
                In the next 4.5.0 release. Soon.
            • #9
              Whoww ! great !

              how do you plan to help people to migrate to this kind of Storage Backend ?
              Meaning... I have a setup in production for more than 2 years now and many files.

              Should i migrate all the data/upload content manually to storage backend or could we have a small script to run that would use these controllers etc.. ?

              Thanks for your support and your good job !!

              Comment

              • #10
                If files will be stored in a single directory w/o subdirectories it will be easy to migrate. You will need to carry over all files from data/upload to your storage, run script in mysql that updates storage column of record in attachment table. And then you will need to write the implementation class for fetching, pushing, removing.

                But first, I need to finish all the work.

                Comment

                • #11
                  To be sure I have understand
                  Currently, we have very big documents like city maps, size is 300 MB for each map. It should be nice to store a document greater than 2 MB (for example) to an external storage provider like dropbox, Google drive, onedrive, ... as thunderbird filelink : https://support.mozilla.org/t5/Basic...nts/ta-p/16289
                  Will this be possible ?

                  Comment

                  • #12
                    We have planned this.

                    Comment

                    • #13
                      Is there anything the community can do to help? I really want this feature and I am willing to contribute if there's a way to speed up the release. I want to start using EspoCRM, but storing files in a local upload folder is a deal breaker because the ephemeral nature of an EC2 instance is very risky.

                      Comment

                      • #14
                        In 4.5.0 release there will be a framework providing an ability to implement external storage. Release is planned for today.

                        Comment

                        • #15
                          Awesome :-))))
                          Thanks for your hard work!

                          Comment

                          Previous 1 2 3 template Next
                          Working...
                          X