Upload all files to an object store

[bandtank]

I see the release is live and the upload feature is listed in the Github release notes. Thanks for your hard work.

[bandtank]

yurikuzn,

Could you please explain how to use the file storage manager feature? I would like to upload files to S3 or Dropbox, but I don't understand if that's possible or how to get started.

[huscmk]

That would indeed be an excellent plus-value if we could externalize media files to Amazon S3 as it would mean no more storage issues on the servers running espocrm.

As I understand, there is now a framework with a storage manager that still needs to get a wrapper for S3 (or any other object storage with an API) but it seems hard to do by the look to that code only as it needs a quite good understanding of how media management is done in espocrm.

Anyone who could help with the task ?

[yuri]

1. In data/config.php need to add parameter 'defaultFileStorage' => 'AmazonS3'


2. Create custom/Espo/Custom/Resources/metadata/app/fileStorage.json with the following content

{
    "implementationClassNameMap": {
        "AmazonS3": "\\Espo\\Custom\\Core\\FileStorage\\Storages\\AmazonS3"
    }
} 

3. Create file custom/Espo/Custom/Core/FileStorage/Storages/AmazonS3.php

<?php

namespace Espo\Custom\Core\FileStorage\Storages;

use \Espo\Entities\Attachment;

use \Espo\Core\Exceptions\Error;

class AmazonS3 extends \Espo\Core\FileStorage\Storages\Base
{
    protected $dependencyList = ['fileManager', 'config'];

    protected function getFileManager()
    {
        return $this->getInjection('fileManager');
    }

    protected function getConfig()
    {
        // in config we can store storage credentials
        return $this->getInjection('config');
    }

    public function unlink(Attachment $attachment)
    {
        // removing file, returns true or false
    }

    public function isFile(Attachment $attachment)
    {
        // checks whether file exists
    }

    public function getContents(Attachment $attachment)
    {
        // returns contents of the file
    }

    public function putContents(Attachment $attachment, $contents)
    {
        // stores contents of the file
    }

    public function getLocalFilePath(Attachment $attachment)
    {
        // returns local file path of copied temporary file to be able to download//
        // if user wants to download the file we need to have it copied on the server as a temporary file
        // maybe this can be improved in future
    }

    public function getDownloadUrl(Attachment $attachment)
    {
        // if there is a direct url to download; likely, leave this method empty
    }

    public function hasDownloadUrl(Attachment $attachment)
    {
        // whether there is a url for download; likely return false;
    }
}

Don't forget to clear cache after changes in metadata.

You can use $attachment->id for filenames on the storage

I'm very busy these days. Don't have time to try to make this implementation.

[huscmk]

Hi, Did anyone completed the AWS S3 implementation by any chance ? Regards,

[bandtank]

I'm very curious about this as well. Being able to upload and download files to S3 would be a huge upgrade to local file storage. It would truly make EspoCRM highly available since the front end could be clustered. This would also be a necessity for Docker containers.

[dabasystem]

HI!

I have a good news and another bad news ...
The good new:
I have managed to put the s3 and it works perfectly: D: D

The bad new:
It only works for me in EspoCrm 4.7 I've tried to do the same configuration in 5.5, 5.4 and I can't get it to work.

Is there a change in the FileStorage? What I can do?

When has it functional I will upload the files in case I can help someone
Thank you so much!

[bandtank]

HI!

I have a good news and another bad news ...
The good new:
I have managed to put the s3 and it works perfectly: D: D

The bad new:
It only works for me in EspoCrm 4.7 I've tried to do the same configuration in 5.5, 5.4 and I can't get it to work.

Is there a change in the FileStorage? What I can do?

When has it functional I will upload the files in case I can help someone
Thank you so much!

This would be absolutely amazing if we can get it to work in 5.5+. Storing files on S3 would completely change my backup and recovery strategy for the better.

[dabasystem]

In the company that we work, we have developed it. You can contact us by email, I guess there will be no problem!
cau@dabasystem.com

[jflores]

dabasystem did you guys get this to work? Any chance you're willing to share the code for the community?

[wtconseil]

In the company that we work, we have developed it. You can contact us by email, I guess there will be no problem!
cau@dabasystem.com

Hi
can you share the code?

[bandtank]

Hi
can you share the code?

Did you ever receive the code? I would very much like to implement this. Storing files to S3 would be huge.

[jflores]

I'm working through this and am stuck trying to instantiate my S3 class.

In my approach, I decided to use a module because I wanted to use Composer to install the Amazon PHP SDK in order to make it "easy" to configure PUTs adn GETs from Amazon S3.

So, here's what I've done.

1. Created a Module Directory

My module is in `application/Espo/Modules/S3storage/FileStorage/Storages/AmazonS3.php`

2. Installed Amazon PHP SDK in application/Espo/Modules/S3Storage
I followed the Amazon documentation here to install the SDK. Once it was installed, I created a file: application/Modules/S3Storage/Resources/autoload.json that referenced the SDK autoload.php:

application/Modules/S3Storage/Resources/autoload.json

{
"AmazonS3": "application/Espo/Modules/S3storage/vendor/autoload.php"
}

3. Added the defaultFileStorage parameter to data/config.php

4. Added the implementationClassNameMap to application/Modules/S3Storage/Resources/metadata/app/fileStorage.json

application/Modules/S3Storage/Resources/metadata/app/fileStorage.json

{
  "implementationClassNameMap": {
  "AmazonS3": "\\Espo\\Modules\\S3storage\\FileStorage\\Storages\\AmazonS3"
  }
}

5. Wrote my AmazonS3.php class

<?php
namespace Espo\Modules\S3storage\FileStorage\Storages;
use \Espo\Entities\Attachment;
use \Espo\Core\Exceptions\Error;
use \Aws\S3\S3Client;
use \Aws\Exception\AwsException;
class AmazonS3 extends \Espo\Core\FileStorage\Storages\Base
{
protected $dependencyList = ['fileManager','config'];
[I]placeholder functions from above[/I]
}

This is where I'm stuck.

For some reason, my AmazonS3 class isn't being instantiated.

I've narrowed down where this breakdown happens to this file:

application/Espo/Core/FileStorage/Manager.php, roughly around line 63. Here's how I'm set up to debug it:

private function getImplementation($storage=null)
{
...
     $className = $this->implementationClassNameMap[$storage];
     $GLOBALS['log']->debug('Here is my $className variable:', [$className, __FILE__,__LINE__]);

// Espo.DEBUG: Here is my $className variable: ["\\Espo\\Modules\\S3storage\\FileStorage\\Storages\\AmazonS3","/var/www/html/espocrm/application/Espo/Core/FileStorage/Manager.php",65] []
// Espo.DEBUG: Here is my $className variable: ["\\Espo\\Core\\FileStorage\\Storages\\EspoUploadDir","/var/www/html/espocrm/application/Espo/Core/FileStorage/Manager.php",65] []

    $implementation = new $className();
    $var = $implementation;
    $GLOBALS['log']->debug('Here is my $implementation variable:', [$var, __FILE__,__LINE__]);
// Espo.DEBUG: Here is my $implementation variable: ["[object] (Espo\\Core\\FileStorage\\Storages\\EspoUploadDir: {})","/var/www/html/espocrm/application/Espo/Core/FileStorage/Manager.php",69] []

}

From the looks of it, the system can 'see' my module class: the implementation map seems to be referencing the files properly. If I create a type (for instance, change "class AmazonS3" to "AmazonS32" it throws an error that it doesn't recognize that class.

But, for whatever reason, it's not instantiating the class at this line:

$implementation = new $className();

And I'm not sure why.

Any thoughts as to why it might not be instantiating?

PS. Once I get this figured out, I"ll post the full implementation details.

[jflores]

Ok, I'm stuck on something that I think should be straightforward, or at least documented...autoloading a file from composer in a module.

Here's my directory structure:

Espo/Modules/S3storage/  
  Core/
     Filestorage/
      Storages/
        AmazonS3.php
  Resources/
    autoload.json
  vendor
    aws/aws-sdk-php/...
    ...other files...
    autoload.php

My autload.json looks like this:

{
  "AmazonS3": "Espo/Modules/S3storage/vendor/autoload.php"
}

and my AmazonS3.php file starts like this:

<?php

namespace Espo\Modules\S3storage\Core\FileStorage\Storages;
use \Espo\Entities\Attachment;
use \Espo\Core\Exceptions\Error;

use Aws\S3\S3Client as S3Client;
use Aws\Exception as AwsException;

class AmazonS3 extends \Espo\Core\FileStorage\Storages\Base
{
  ...
}

The error I keep getting is:

Espo.ERROR: Uncaught Exception Error: "Class 'aws\S3\S3Client' not found"

And here is my autoload_psr4.php file in my module "vendor/composer"

<?php

// autoload_psr4.php @generated by Composer

$vendorDir = dirname(dirname(__FILE__));
$baseDir = dirname($vendorDir);

return array(
    'Symfony\\Polyfill\\Php72\\' => array($vendorDir . '/symfony/polyfill-php72'),
    'Symfony\\Polyfill\\Mbstring\\' => array($vendorDir . '/symfony/polyfill-mbstring'),
    'Symfony\\Polyfill\\Intl\\Idn\\' => array($vendorDir . '/symfony/polyfill-intl-idn'),
    'Psr\\Http\\Message\\' => array($vendorDir . '/psr/http-message/src'),
    'JmesPath\\' => array($vendorDir . '/mtdowling/jmespath.php/src'),
    'GuzzleHttp\\Psr7\\' => array($vendorDir . '/guzzlehttp/psr7/src'),
    'GuzzleHttp\\Promise\\' => array($vendorDir . '/guzzlehttp/promises/src'),
    'GuzzleHttp\\' => array($vendorDir . '/guzzlehttp/guzzle/src'),
    'Aws\\' => array($vendorDir . '/aws/aws-sdk-php/src'),
);

Based on the Amazon documentation, I'm pretty sure I'm calling the namespaces correctly. However, if someone could let me know if I'm loading the aws library properly given my directory structure, that would be hugely helpful.

[huscmk]

Hi,
Any luck with your implementation?

Regards

[bandtank]

Has anyone made any progress on this? It would be a huge enhancement for my environment. I really want to put my CRM files somewhere other than the container or host.

[jflores]

huscmk bandtank not yet. still working on it when I get time. I'll post a solution as soon as I figure it out.

[agarwalmayank027]

has anyone been able to make progress on this?