Announcement

Collapse
No announcement yet.

Upload all files to an object store

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    I see the release is live and the upload feature is listed in the Github release notes. Thanks for your hard work.

    Comment


    • #17
      yurikuzn,

      Could you please explain how to use the file storage manager feature? I would like to upload files to S3 or Dropbox, but I don't understand if that's possible or how to get started.

      Comment


      • huscmk
        huscmk commented
        Editing a comment
        That would indeed be an excellent plus-value if we could externalize media files to Amazon S3 as it would mean no more storage issues on the servers running espocrm.

        As I understand, there is now a framework with a storage manager that still needs to get a wrapper for S3 (or any other object storage with an API) but it seems hard to do by the look to that code only as it needs a quite good understanding of how media management is done in espocrm.

        Anyone who could help with the task ?

    • #18
      1. In data/config.php need to add parameter 'defaultFileStorage' => 'AmazonS3'


      2. Create custom/Espo/Custom/Resources/metadata/app/fileStorage.json with the following content
      PHP Code:
      {
          
      "implementationClassNameMap": {
              
      "AmazonS3""\\Espo\\Custom\\Core\\FileStorage\\Storages\\AmazonS3"
          
      }


      3. Create file custom/Espo/Custom/Core/FileStorage/Storages/AmazonS3.php

      PHP Code:
      <?php

      namespace Espo\Custom\Core\FileStorage\Storages;

      use 
      \Espo\Entities\Attachment;

      use 
      \Espo\Core\Exceptions\Error;

      class 
      AmazonS3 extends \Espo\Core\FileStorage\Storages\Base
      {
          protected 
      $dependencyList = ['fileManager''config'];

          protected function 
      getFileManager()
          {
              return 
      $this->getInjection('fileManager');
          }

          protected function 
      getConfig()
          {
              
      // in config we can store storage credentials
              
      return $this->getInjection('config');
          }

          public function 
      unlink(Attachment $attachment)
          {
              
      // removing file, returns true or false
          
      }

          public function 
      isFile(Attachment $attachment)
          {
              
      // checks whether file exists
          
      }

          public function 
      getContents(Attachment $attachment)
          {
              
      // returns contents of the file
          
      }

          public function 
      putContents(Attachment $attachment$contents)
          {
              
      // stores contents of the file
          
      }

          public function 
      getLocalFilePath(Attachment $attachment)
          {
              
      // returns local file path of copied temporary file to be able to download//
              // if user wants to download the file we need to have it copied on the server as a temporary file
              // maybe this can be improved in future
          
      }

          public function 
      getDownloadUrl(Attachment $attachment)
          {
              
      // if there is a direct url to download; likely, leave this method empty
          
      }

          public function 
      hasDownloadUrl(Attachment $attachment)
          {
              
      // whether there is a url for download; likely return false;
          
      }
      }
      Don't forget to clear cache after changes in metadata.

      You can use $attachment->id for filenames on the storage

      I'm very busy these days. Don't have time to try to make this implementation.
      Last edited by yuri; 03-14-2017, 03:37 PM.

      Comment


      • #19
        Hi, Did anyone completed the AWS S3 implementation by any chance ? Regards,

        Comment


        • #20
          I'm very curious about this as well. Being able to upload and download files to S3 would be a huge upgrade to local file storage. It would truly make EspoCRM highly available since the front end could be clustered. This would also be a necessity for Docker containers.

          Comment


          • #21
            HI!

            I have a good news and another bad news ...
            The good new:
            I have managed to put the s3 and it works perfectly: D: D

            The bad new:
            It only works for me in EspoCrm 4.7 I've tried to do the same configuration in 5.5, 5.4 and I can't get it to work.

            Is there a change in the FileStorage? What I can do?

            When has it functional I will upload the files in case I can help someone
            Thank you so much!

            Comment


            • #22
              Originally posted by dabasystem View Post
              HI!

              I have a good news and another bad news ...
              The good new:
              I have managed to put the s3 and it works perfectly: D: D

              The bad new:
              It only works for me in EspoCrm 4.7 I've tried to do the same configuration in 5.5, 5.4 and I can't get it to work.

              Is there a change in the FileStorage? What I can do?

              When has it functional I will upload the files in case I can help someone
              Thank you so much!
              This would be absolutely amazing if we can get it to work in 5.5+. Storing files on S3 would completely change my backup and recovery strategy for the better.

              Comment


              • #23
                In the company that we work, we have developed it. You can contact us by email, I guess there will be no problem!
                cau@dabasystem.com

                Comment


                • jflores
                  jflores commented
                  Editing a comment
                  dabasystem did you guys get this to work? Any chance you're willing to share the code for the community?

              • #24
                Originally posted by dabasystem View Post
                In the company that we work, we have developed it. You can contact us by email, I guess there will be no problem!
                cau@dabasystem.com
                Hi
                can you share the code?

                Comment


                • #25
                  Originally posted by wtconseil View Post

                  Hi
                  can you share the code?
                  Did you ever receive the code? I would very much like to implement this. Storing files to S3 would be huge.

                  Comment


                  • #26
                    I'm working through this and am stuck trying to instantiate my S3 class.

                    In my approach, I decided to use a module because I wanted to use Composer to install the Amazon PHP SDK in order to make it "easy" to configure PUTs adn GETs from Amazon S3.

                    So, here's what I've done.

                    1. Created a Module Directory

                    My module is in `application/Espo/Modules/S3storage/FileStorage/Storages/AmazonS3.php`

                    2. Installed Amazon PHP SDK in application/Espo/Modules/S3Storage
                    I followed the Amazon documentation here to install the SDK. Once it was installed, I created a file: application/Modules/S3Storage/Resources/autoload.json that referenced the SDK autoload.php:

                    application/Modules/S3Storage/Resources/autoload.json
                    Code:
                    {
                    "AmazonS3": "application/Espo/Modules/S3storage/vendor/autoload.php"
                    }
                    3. Added the defaultFileStorage parameter to data/config.php

                    4. Added the implementationClassNameMap to application/Modules/S3Storage/Resources/metadata/app/fileStorage.json

                    application/Modules/S3Storage/Resources/metadata/app/fileStorage.json
                    Code:
                    {
                      "implementationClassNameMap": {
                      "AmazonS3": "\\Espo\\Modules\\S3storage\\FileStorage\\Storages\\AmazonS3"
                      }
                    }
                    5. Wrote my AmazonS3.php class

                    Code:
                    <?php
                    namespace Espo\Modules\S3storage\FileStorage\Storages;
                    use \Espo\Entities\Attachment;
                    use \Espo\Core\Exceptions\Error;
                    use \Aws\S3\S3Client;
                    use \Aws\Exception\AwsException;
                    class AmazonS3 extends \Espo\Core\FileStorage\Storages\Base
                    {
                    protected $dependencyList = ['fileManager','config'];
                    [I]placeholder functions from above[/I]
                    }
                    This is where I'm stuck.

                    For some reason, my AmazonS3 class isn't being instantiated.

                    I've narrowed down where this breakdown happens to this file:

                    application/Espo/Core/FileStorage/Manager.php, roughly around line 63. Here's how I'm set up to debug it:

                    Code:
                    private function getImplementation($storage=null)
                    {
                    ...
                         $className = $this->implementationClassNameMap[$storage];
                         $GLOBALS['log']->debug('Here is my $className variable:', [$className, __FILE__,__LINE__]);
                    
                    // Espo.DEBUG: Here is my $className variable: ["\\Espo\\Modules\\S3storage\\FileStorage\\Storages\\AmazonS3","/var/www/html/espocrm/application/Espo/Core/FileStorage/Manager.php",65] []
                    // Espo.DEBUG: Here is my $className variable: ["\\Espo\\Core\\FileStorage\\Storages\\EspoUploadDir","/var/www/html/espocrm/application/Espo/Core/FileStorage/Manager.php",65] []
                    
                        $implementation = new $className();
                        $var = $implementation;
                        $GLOBALS['log']->debug('Here is my $implementation variable:', [$var, __FILE__,__LINE__]);
                    // Espo.DEBUG: Here is my $implementation variable: ["[object] (Espo\\Core\\FileStorage\\Storages\\EspoUploadDir: {})","/var/www/html/espocrm/application/Espo/Core/FileStorage/Manager.php",69] []
                    
                    }
                    From the looks of it, the system can 'see' my module class: the implementation map seems to be referencing the files properly. If I create a type (for instance, change "class AmazonS3" to "AmazonS32" it throws an error that it doesn't recognize that class.

                    But, for whatever reason, it's not instantiating the class at this line:

                    Code:
                    $implementation = new $className();
                    And I'm not sure why.

                    Any thoughts as to why it might not be instantiating?

                    PS. Once I get this figured out, I"ll post the full implementation details.

                    Comment


                    • #27
                      Ok, I'm stuck on something that I think should be straightforward, or at least documented...autoloading a file from composer in a module.

                      Here's my directory structure:

                      Code:
                      Espo/Modules/S3storage/  
                        Core/
                           Filestorage/
                            Storages/
                              AmazonS3.php
                        Resources/
                          autoload.json
                        vendor
                          aws/aws-sdk-php/...
                          ...other files...
                          autoload.php
                      My autload.json looks like this:
                      Code:
                      {
                        "AmazonS3": "Espo/Modules/S3storage/vendor/autoload.php"
                      }
                      and my AmazonS3.php file starts like this:

                      Code:
                      <?php
                      
                      namespace Espo\Modules\S3storage\Core\FileStorage\Storages;
                      use \Espo\Entities\Attachment;
                      use \Espo\Core\Exceptions\Error;
                      
                      use Aws\S3\S3Client as S3Client;
                      use Aws\Exception as AwsException;
                      
                      class AmazonS3 extends \Espo\Core\FileStorage\Storages\Base
                      {
                        ...
                      }
                      The error I keep getting is:

                      Code:
                      Espo.ERROR: Uncaught Exception Error: "Class 'aws\S3\S3Client' not found"
                      And here is my autoload_psr4.php file in my module "vendor/composer"

                      Code:
                      <?php
                      
                      // autoload_psr4.php @generated by Composer
                      
                      $vendorDir = dirname(dirname(__FILE__));
                      $baseDir = dirname($vendorDir);
                      
                      return array(
                          'Symfony\\Polyfill\\Php72\\' => array($vendorDir . '/symfony/polyfill-php72'),
                          'Symfony\\Polyfill\\Mbstring\\' => array($vendorDir . '/symfony/polyfill-mbstring'),
                          'Symfony\\Polyfill\\Intl\\Idn\\' => array($vendorDir . '/symfony/polyfill-intl-idn'),
                          'Psr\\Http\\Message\\' => array($vendorDir . '/psr/http-message/src'),
                          'JmesPath\\' => array($vendorDir . '/mtdowling/jmespath.php/src'),
                          'GuzzleHttp\\Psr7\\' => array($vendorDir . '/guzzlehttp/psr7/src'),
                          'GuzzleHttp\\Promise\\' => array($vendorDir . '/guzzlehttp/promises/src'),
                          'GuzzleHttp\\' => array($vendorDir . '/guzzlehttp/guzzle/src'),
                          'Aws\\' => array($vendorDir . '/aws/aws-sdk-php/src'),
                      );
                      Based on the Amazon documentation, I'm pretty sure I'm calling the namespaces correctly. However, if someone could let me know if I'm loading the aws library properly given my directory structure, that would be hugely helpful.
                      Last edited by jflores; 05-09-2020, 04:11 AM.

                      Comment


                      • huscmk
                        huscmk commented
                        Editing a comment
                        Hi,
                        Any luck with your implementation?

                        Regards

                    • #28
                      Has anyone made any progress on this? It would be a huge enhancement for my environment. I really want to put my CRM files somewhere other than the container or host.

                      Comment


                      • #29
                        huscmk bandtank not yet. still working on it when I get time. I'll post a solution as soon as I figure it out.

                        Comment


                        • #30
                          has anyone been able to make progress on this?

                          Comment

                          Working...
                          X