Upload all files to an object store

[rinorway]

In the meantime you might try rclone with mount option

https://rclone.org/commands/rclone_mount/

[Laimonas]

Thanks, I will try to implement the second option!

[yuri]

Hi,

1. No. But it can be easily implemented by a developer.

2. You need to add some logic (it can in BeforeSave hook for Attachment) that sets different 'storage' attribute ('EspoUploadDir' or 'AwsS3') depending on conditions. An attachment record has the 'storage' attribute that indicates what storage implementation to use.

[Laimonas]

Hi,

How is it going with AWS S3 to you guys? It seems to be working, but I have a few questions:

1. Are you capable to separate diverse attachments? Let's say I upload .png pictures as "Attachment Multiple" (field - attachments, type - image/png) and .pdf documents as "File" (field - file, type - application/pdf) and they are all uploaded to AWS S3 in the same bucket. Is it possible to upload them to different buckets? I want to set different lifecycle rules to .png and .pdf files (I want to delete .png pictures after 180 days, but not the .pdf contracts with clients). Do you have any ideas on how could I implement that?

2. Is it possible not to send "File" type attachments to AWS, but keep them in the CRM data/upload folder?

[emillod]

https://github.com/espocrm/espocrm/issues/1934

[agarwalmayank027]

has anyone been able to make progress on this?

[jflores]

huscmk bandtank not yet. still working on it when I get time. I'll post a solution as soon as I figure it out.

[bandtank]

Has anyone made any progress on this? It would be a huge enhancement for my environment. I really want to put my CRM files somewhere other than the container or host.

[huscmk]

Hi,
Any luck with your implementation?

Regards

[jflores]

Ok, I'm stuck on something that I think should be straightforward, or at least documented...autoloading a file from composer in a module.

Here's my directory structure:

Espo/Modules/S3storage/  
  Core/
     Filestorage/
      Storages/
        AmazonS3.php
  Resources/
    autoload.json
  vendor
    aws/aws-sdk-php/...
    ...other files...
    autoload.php

My autload.json looks like this:

{
  "AmazonS3": "Espo/Modules/S3storage/vendor/autoload.php"
}

and my AmazonS3.php file starts like this:

<?php

namespace Espo\Modules\S3storage\Core\FileStorage\Storages;
use \Espo\Entities\Attachment;
use \Espo\Core\Exceptions\Error;

use Aws\S3\S3Client as S3Client;
use Aws\Exception as AwsException;

class AmazonS3 extends \Espo\Core\FileStorage\Storages\Base
{
  ...
}

The error I keep getting is:

Espo.ERROR: Uncaught Exception Error: "Class 'aws\S3\S3Client' not found"

And here is my autoload_psr4.php file in my module "vendor/composer"

<?php

// autoload_psr4.php @generated by Composer

$vendorDir = dirname(dirname(__FILE__));
$baseDir = dirname($vendorDir);

return array(
    'Symfony\\Polyfill\\Php72\\' => array($vendorDir . '/symfony/polyfill-php72'),
    'Symfony\\Polyfill\\Mbstring\\' => array($vendorDir . '/symfony/polyfill-mbstring'),
    'Symfony\\Polyfill\\Intl\\Idn\\' => array($vendorDir . '/symfony/polyfill-intl-idn'),
    'Psr\\Http\\Message\\' => array($vendorDir . '/psr/http-message/src'),
    'JmesPath\\' => array($vendorDir . '/mtdowling/jmespath.php/src'),
    'GuzzleHttp\\Psr7\\' => array($vendorDir . '/guzzlehttp/psr7/src'),
    'GuzzleHttp\\Promise\\' => array($vendorDir . '/guzzlehttp/promises/src'),
    'GuzzleHttp\\' => array($vendorDir . '/guzzlehttp/guzzle/src'),
    'Aws\\' => array($vendorDir . '/aws/aws-sdk-php/src'),
);

Based on the Amazon documentation, I'm pretty sure I'm calling the namespaces correctly. However, if someone could let me know if I'm loading the aws library properly given my directory structure, that would be hugely helpful.

[jflores]

I'm working through this and am stuck trying to instantiate my S3 class.

In my approach, I decided to use a module because I wanted to use Composer to install the Amazon PHP SDK in order to make it "easy" to configure PUTs adn GETs from Amazon S3.

So, here's what I've done.

1. Created a Module Directory

My module is in `application/Espo/Modules/S3storage/FileStorage/Storages/AmazonS3.php`

2. Installed Amazon PHP SDK in application/Espo/Modules/S3Storage
I followed the Amazon documentation here to install the SDK. Once it was installed, I created a file: application/Modules/S3Storage/Resources/autoload.json that referenced the SDK autoload.php:

application/Modules/S3Storage/Resources/autoload.json

{
"AmazonS3": "application/Espo/Modules/S3storage/vendor/autoload.php"
}

3. Added the defaultFileStorage parameter to data/config.php

4. Added the implementationClassNameMap to application/Modules/S3Storage/Resources/metadata/app/fileStorage.json

application/Modules/S3Storage/Resources/metadata/app/fileStorage.json

{
  "implementationClassNameMap": {
  "AmazonS3": "\\Espo\\Modules\\S3storage\\FileStorage\\Storages\\AmazonS3"
  }
}

5. Wrote my AmazonS3.php class

<?php
namespace Espo\Modules\S3storage\FileStorage\Storages;
use \Espo\Entities\Attachment;
use \Espo\Core\Exceptions\Error;
use \Aws\S3\S3Client;
use \Aws\Exception\AwsException;
class AmazonS3 extends \Espo\Core\FileStorage\Storages\Base
{
protected $dependencyList = ['fileManager','config'];
[I]placeholder functions from above[/I]
}

This is where I'm stuck.

For some reason, my AmazonS3 class isn't being instantiated.

I've narrowed down where this breakdown happens to this file:

application/Espo/Core/FileStorage/Manager.php, roughly around line 63. Here's how I'm set up to debug it:

private function getImplementation($storage=null)
{
...
     $className = $this->implementationClassNameMap[$storage];
     $GLOBALS['log']->debug('Here is my $className variable:', [$className, __FILE__,__LINE__]);

// Espo.DEBUG: Here is my $className variable: ["\\Espo\\Modules\\S3storage\\FileStorage\\Storages\\AmazonS3","/var/www/html/espocrm/application/Espo/Core/FileStorage/Manager.php",65] []
// Espo.DEBUG: Here is my $className variable: ["\\Espo\\Core\\FileStorage\\Storages\\EspoUploadDir","/var/www/html/espocrm/application/Espo/Core/FileStorage/Manager.php",65] []

    $implementation = new $className();
    $var = $implementation;
    $GLOBALS['log']->debug('Here is my $implementation variable:', [$var, __FILE__,__LINE__]);
// Espo.DEBUG: Here is my $implementation variable: ["[object] (Espo\\Core\\FileStorage\\Storages\\EspoUploadDir: {})","/var/www/html/espocrm/application/Espo/Core/FileStorage/Manager.php",69] []

}

From the looks of it, the system can 'see' my module class: the implementation map seems to be referencing the files properly. If I create a type (for instance, change "class AmazonS3" to "AmazonS32" it throws an error that it doesn't recognize that class.

But, for whatever reason, it's not instantiating the class at this line:

$implementation = new $className();

And I'm not sure why.

Any thoughts as to why it might not be instantiating?

PS. Once I get this figured out, I"ll post the full implementation details.

[jflores]

dabasystem did you guys get this to work? Any chance you're willing to share the code for the community?

[bandtank]

Hi
can you share the code?

Did you ever receive the code? I would very much like to implement this. Storing files to S3 would be huge.

[wtconseil]

In the company that we work, we have developed it. You can contact us by email, I guess there will be no problem!
cau@dabasystem.com

Hi
can you share the code?

[dabasystem]

In the company that we work, we have developed it. You can contact us by email, I guess there will be no problem!
cau@dabasystem.com