Chunk Upload - AWS S3

[zkuun8]

Hello,
I have configured FileStorage as AWS S3, using a third-party provider.
I increased the limits in espocrm.ini and data/config.php to allow files up to 30GB.

I noticed that when the Upload Chunk Size (Mb) is set to 5, all files smaller than 5MB are stored in "AWS" and all files larger than 5MB are stored in "EspoUploadDir."
When I change the Upload Chunk Size (Mb) to 10, all files smaller than 10MB go to "AWS," and all files larger than 10MB go to "EspoUploadDir."

Is the issue coming from my AWS S3 provider (Vultr), from EspoCRM’s AWS functionality, or somewhere else?

1. Log:
   1. Espo log in data/logs directory : Empty
   2. web server error log : ?? Not able to check
   3. browser console (F12 key) : Empty
2. EspoCRM version : 9.0.1
3. PHP 8.2.24
4. Database MariaDB 10.5.27

[yuri]

Hi,

Did you check bit later? After all chunks are uploaded, the file is supposed to moved to the S3 storage by a cron job.

[yuri]

Then, the locally stored file is removed after 3 hours.

[yuri]

We ask (in bug report guidelines) to provide more info in bug report posts. Knowing if there are any errors in logs or no errors would have been helpful.

[zkuun8]

I've edited my initial post with detail asked in guidelines

[zkuun8]

Hi Yuri,
1) there is no log error
2) I've upload an attachment yesterday which is still in EspoUploadDir

[zkuun8]

Upload Chunk Size (Mb) is the important point.
It seems when the value is higher than the size of the file, it upload to the AWS, but when the value is lower than the file, it uploads to EspoUploadDir
That's why I wonder if there is an issue with chunk upload feature, or if it is from the S3 provider that may not manage chunk as expected, or an issue from the serv..

[yuri]

Do you have jobs properly running in your Espo instance? You can check at Administration > Jobs.

[zkuun8]

and there an exemple of a file uploaded directly to AWS (because its size est lower than the param Upload Chunk Size (Mb))

[zkuun8]

ok, it seems it s pending

[yuri]

Please go to Administration > Jobs.

Then, in the search box enter:

*MoveToStorage

And click the Search button.

Does it show any jobs? What status of these jobs?

[yuri]

It seems jobs are not running. Maybe your cron stopped yesterday?

[yuri]

On the Administration page, when the cron is not running for a while, there should be the red colored message in the top-right corner.

[yuri]

Or the cron is disabled in Espo. Administration > Settings > Disable Cron. Should be unchecked.

[item]

Hi Yuri,
for one test with a provider, i have something issue with cron.. (not with AWS3 or else)
look cron jobs "createdAt" ... it's variable time interval .. no respect certainly "the scheduled period"..
so, i think some "low cost provider" do something with cron.

Maybe.

[zkuun8]

Ok, so,

I had to reactivate Cron it was not running for some reason (thanks for the tips), but there is still an issue :

Then, jobs was running and then it fails : [2025-01-25 21:51:19] CRITICAL: (0) Failed job 67953c7af33995a7c. Unable to write file at location: 67953c69bf6963141. Could not contact remote server. :: /var/www/html/application/Espo/Core/FileStorage/Storages/AwsS3.php(151)

For some files, it works (if their size is lower than Upload Chunk Size (Mb)

[yuri]

I suspect that PHP cannot connect to a remote server when it runs from CLI (cron). But can when it runs though the webserver (when you upload without chunks).