Document Store Directory Structure

[seandakid]

Can the document management portion of ESPOCRM have a specific directory structure? For instance, we have a folder structure currently that includes folders with physical addresses of customers and inside those folders are a number of documents. If we import our document store into ESPO, will we be able to search based on that folders or is it just a flat file system?

Thanks for your time.

[item]

Hi,
i try to respond with my skill (!) and experiment and my english:

actually, there are 2 type of ... interface.
- out-of-box : data/upload/
- AWS3 ? ... never use, so can help.

for out-of-box, all is "yes" flat folder .. all is data/upload/{$id}

you can write your own "system folder storage" .. but need rewrite code for many. (api/download/...)

i don't know where you will store "pathToFolder" .. but big work.

Yuri can correct me.. i think not easy to do, need many work.

Good luck

[seandakid]

Thanks for the info item!

[Firyo]

Hi there,

You can store your "Attachment" in custom "FileStorage" that will extends the default EspoCRM storage (Espo\Core\FileStorage\Storages\EspoUploadDir)
Like item said, by default every file is stored in the "EspoUploadDir" which sets the files in the "data/upload/" folder.

The type of file storage used can be different for each "Attachment" you create (or import).
It is based on the "storage" field in the "Attachment" SQL table.

Here is an example (EspoCRM 7.1.11 in my case) :

File path : "YourCustomModule/Core/FileStorage/Storages/YourCustomFileStorageName"

File path : "YourCustomModule/Resources/metadata/app/fileStorage.json"



Documentation articles here :

https://docs.espocrm.com/administration/file-storage/

https://docs.espocrm.com/development/metadata/app-file-storage/

Regards,
Firyo.

[seandakid]

Awesome! Thanks for the details Firyo. I will play around with it.

[yuri]

I recommend to re-implement the class fully, without extending. As interfaces are unlikely to be changed in future, but protected method is likely to be changed.

[item]

Hi,
my understand of request of seandakid was a directory structure.
as Firyo have proposed solution.. my post is for brainstorm for new feature

for smalBusiness with 1000 attachments ?
what about business with 100000 attachments ? in one folder ?
try to open a folder with 100000 attachments ?

i think, we need to do like "Apple foto".. store by date.

My feature request will be so :
in install of espocrm : select if "folderByDate"... or not.
if attachment is created "2023-09-29" .. put in folder "2023/09"
if attachment is created "2023-10-01" .. put in folder "2023/10"
so all is in folder by "Y/m"
i think another structure will be very hard to "implement".
Don't forget if you have email... all is stored too.

There are positive or negative... maybe ... to brainstorm and feature request or not

[yuri]

BTW, I would not consider 100,000 files as big.

[shalmaxb]

I agree with item, that it could be worthwhile to think about modifications in storing attachments. I am aware, that the way espoCRM does this right now, is well thought and has lots of advantages. But is also has disadvantages. For me this means:

1. as item said, it is very difficult to handle one big folder with a big amount of data. Making backups can be a nightmare.
2. I see the reason for renaming every attachment with a generated number and without file extension. From the database point of view, this is perhaps the only way to handle attachments. But there are cases, wher it is a bit of a problem. I have broken images from time to time, which I have to upload again, but the broken image persists in the upload folder. It is quite impossible to find such an image to be able to delete it.
3. I know it is possible to store attachments in a cloud storage, but realizing this for any other cloud than AWS is difficult for one, who is not skilled in coding. So a more easy possibility would be appreciated.
4. separating into folders by date seems to be the best viable solution, when building a folder structure, which would be appropriate to any kind of use case.

[yuri]

> I have broken images from time to time, which I have to upload again, but the broken image persists in the upload folder. It is quite impossible to find such an image to be able to delete it.

You can find and delete a specific attachment at Administration > Attachments.

> it is very difficult to handle one big folder with a big amount of data. Making backups can be a nightmare.

What exactly is the problem? For backups, if you don't want to backup old files, to have the archive smaller, you can use a command that archives only files created after a specific date.

One can create a custom storage implementation that will get the date from "createdAt" and store in a corresponding folder.

I've never had the need to look into the data/upload folder on the production. The same as we don't care how the database is storing data, and how the file system is storing files on the disk.

[shalmaxb]

yuri, my post is by no means any criticism on how espoCRM is programmed and how solutions are implemented. It had been only my thoughts, working with this - honestly - wonderful software.
I admit, I never looked into that Admin function attachments, it is merely one thing of the many features, that I did not explore yet.
I already make differential backups, but I already once had to migrate a by my fault corrupted installation to a new server. As I use many images in my application (what is due to the fact, that espoCRM suits for so many purposes), my upload folder reaches very soon gigabyte dimensions. This is the only reason for a big folder.

You gave us "amateurs" such a great software, with which I am able to create something, I never had dreamed of before. Unfortunately you did not count with the many ideas, that are possible to be realized, I guess. And we are constructing things, that go beyond our knowledge sometimes.
Because of that I struggle, I learn, I get help and I can achieve something. Sometimes this comes with problems, for which you are not responsible but I appreciate that those can be discussed here.
Thank you again and nothing for bad.

[yuri]

The custom storage engine that stores to directories based on the created-at date field is an easy task for a developer. The system provides the interface and config param. I'm sure there are developers here on the forum who could accomplish this task (maybe in a matter of minutes).

Adding it to base Espo? I'm not sure we really need it. Adding something to the product is a more complex thing than implementing the same as a 3rd party extensions. We have many tasks waiting to be tackled, it's too much accumulated for years.

[shalmaxb]

Is it true then, that I can delete filtered orphans from the attachments in admin? If yes, that is really a great feature, which I did not know.

[yuri]

Yes, you can. With 'Select all result' should work too. You can also sort by file size to inspect the biggest files, filter by date.

[yuri]

Make sure you have a backup before doing a big clean-up. Just in case.

[item]

Hi Yuri,

if you read .. thanks for info for make a custom storage. Yes, it's seemts easy (for me in this case).

just one question if you respond

do you think it's possible to have "easy way? my skill" only email attachment in a custom other folder than the normal folder.
in other word, email (html/logo/attachment) , i wil save in other folder than the storage folder.

Best Regards

EDIT : custom folder for email (image/logo/...) certainly very hard to implement, so my question is not more a question

[yuri]

It's both easy and not. You can check an attachment's parentType if it == 'Email'. But we have the attachment cloning ability. E.g. an attachment can be duplicated and related to another entity w/o copying the file. So there are two attachments pointing to the same file. One is related to an email another to a document, for example. It's not that trivial.