Request for Mass Import of 200 Documents into Documents Entity

[sapyantuk]

Dear Developers,

I hope this message finds you well. I am reaching out to discuss an important task that requires your expertise. Currently, I am faced with the challenge of uploading a significant number of documents, specifically over 200, into the Documents Entity.

As you can imagine, manually uploading each document one by one would be a time-consuming and laborious process. Hence, I am eager to explore alternative methods that could streamline this task and improve efficiency. In this regard, I am wondering if there is a way to perform a mass import of these 200 documents.

[item]

Hi,
need développer skill for me and script that loop over a directory and post attachment and post document

https://github.com/espocrm/documentation/blob/master/docs/development/api/attachment.md#uploading

[rabii]

hey sapyantuk

Just in case you still need a tool to help you upload the files to the server, please find attached a small php app which will allow you to do that, the way it works if that you unzip the file and just drop it to a php web server and access the app folder, when you access the web folder you should be able to modify index.php (add your crm domain and your api key) once that is done, then you can access the application and provide the full path to the directory where the files exist, and press upload the files. the application will scan the files directory and get rid of any . or .. files and then will loop through the files and send a post request to create an attachment and get returned id and after that send another post request to create a document and related the attachment to the document.

i would advise you to try this out with one or two files and once you are happy with the result then do it for all 200 files.

here is a video that shows how it works.

https://player.vimeo.com/video/827984840?h=921310ab39&badge=0&autopause=0&player_id=0&app_id=58479

I hope this helps.

[sapyantuk]

Thank you, rabii , for your response.

I have files on my system organized within folders, and I would like to upload these files while preserving the folder structure. For example, I have a file named "abc.docx" located inside a folder called "Abc". Is there a way to achieve uploading these files with the same folder structure intact?

[rabii]

It is possible to apply such logic, it will just add necessity to scan subfolder and create them in the process, let me see if i can update the code to add this functionality for you.

[esforim]

Wow I thought this was something hard to use but looking at the video it so easy!

And lol these filler time in video:

[rabii]

i was doing the video while working on something else so i was in rush and have few missed typing 😁😅

Eevrything is possible with Espocrm mate, i am just diving into some cool staff you can do with the system.

[sapyantuk]

rabii any update on uploading files with the same folder structure ?

[rabii]

sapyantuk

Sorry i was busy with some urgent work. i have modified the code to loop through a directory and get the subdirectories and then create documentfolder record via post request, after that loop through each subdirectory and create attachment, document and link them - also document will be linked with the correct directory. i have not had time to properly display any errors but just echo them into the screen if there are any errors.

Make sure your api user has the permission to create DocumentFolder and Document in their role. Just provide the full path to the directory (main directory) and press upload files button. it might take some times if the folders and files are big in size.

Please test this out with few files before applying this on the 200 pdf files. also if you have any images / audio etc make sure the file field in documents accept those extensions.

find attached modified version.

Thanks

[esforim]

Do you have a git or somewhere I can automatic stalk you

[sapyantuk]

rabii


Dear Rabii,

I hope this message finds you well. I wanted to inform you that I have successfully uploaded the file you provided. Additionally, I have made the necessary changes by replacing the ESPo CRM URL and the API code in the "index.php" file.

However, I have encountered an issue when entering the local system's full path. It consistently displays an error message stating "Invalid path." Furthermore, on some occasions, I am confronted with an HTTP ERROR 500.

I would greatly appreciate your assistance in resolving this matter. Please let me know if you require any further information or if there are any specific steps I should take to address this issue.

Thank you for your attention to this matter.

[rabii]

Hey
"Invalid path." means that the path that you provided is not correct it should be something like (C:\main-folder\documents-folder) on mac you could use the pwd on the main folder to get the full path.

HTTP ERROR 500 Means some resources / records were not created (this could be cause by different issues e.g the file extension is not permitted on the file on your entity).

do you check the crm to see if all necessary folders / documents and attachments are created?

[sapyantuk]

The path I entered is as follows: C:\Users\Auditpcmain\Desktop\folder. However, upon entering this path, I received an error message stating "Invalid path."

To further investigate this issue, I have checked the CRM system, and it appears that nothing has been created or associated with the provided path. It seems that there might be a problem with the path itself or with the system's configuration.

I kindly request your assistance in resolving this matter. Please let me know if you require any additional information or if there are any specific steps I should take to troubleshoot the path issue.

[sapyantuk]

rabii
The path I entered is as follows: C:\Users\Auditpcmain\Desktop\folder. However, upon entering this path, I received an error message stating "Invalid path."

To further investigate this issue, I have checked the CRM system, and it appears that nothing has been created or associated with the provided path. It seems that there might be a problem with the path itself or with the system's configuration.

I kindly request your assistance in resolving this matter. Please let me know if you require any additional information or if there are any specific steps I should take to troubleshoot the path issue.

[rabii]

does it say (Invalid directory path) ?

[sapyantuk]

Yes. Sometimes , i couldnt reproduce all time . sometime it says Invalid directory path and most of the time HTTP ERROR 500
If you are available please join teamviewer for 2 min
822 816 264

[rabii]

The error 500 is the returned error because no record was being able to be created. i have noticed that you have a wrong path if you are on windows it should (C:/Users/Auditpcmain/Desktop/folder) otherwise the it will trigger an error because php will not be able to read the path correctly. use the forward slash.

This should be the correct path C:/Users/Auditpcmain/Desktop/folder

[sapyantuk]

rabii i tried

• • C:/Users/Auditpcmain/Desktop/folder

still same issue

[rabii]

as mentioned before please try to test this on one folder and one file because you proceed with the 200 files as this could cause a problem of memory limit and therefore it will cause issues.

[sapyantuk]

7q92r8us


this is the password

[rabii]

what teamview password

[sapyantuk]

7q92r8us


password

[sapyantuk]

rabii


I have installed a local environment called XAMPP and attempted to use it, but I am still encountering an error. I have verified that the cURL extension is installed, yet the error persists.

[sapyantuk]

rabii

I hope this message finds you well. I wanted to inform you that I have successfully installed the local environment that you suggested, and as a result, the application has started working. However, I encountered an error that I'd like to discuss.

The issue arises when we upload a document through the portal. Although records are being created, the problem lies in opening or downloading the file itself. It appears that the size of the file is consistently shown as 0 KB, indicating that there is no content within it.

Upon conducting a thorough analysis of the issue, I discovered the root cause. The files are being scanned from the input path as expected, but the code seems to be searching for these files within the home directory where the application is hosted. This discrepancy in the search locations is likely the reason behind the empty files.

Additionally, I wanted to bring to your attention another issue that I've come across. When we create or send files through the application, it seems that the files are not being posted in the same folder that already exists in Espo.

To illustrate, let's say there is a folder in Espo called "Account." When I have some files on my local computer and attempt to upload them, the API creates a new folder named "Account" instead of posting the files in the existing directory in Espo.

I was wondering if it would be possible to implement a check for the folder name. If the folder name already exists, we can then post the files in the existing directory in Espo. This would allow us to upload the documents in our desired structure by leveraging the same folder structure that exists in Espo.

Combining the resolution of the file size issue and the implementation of the folder check feature would greatly improve the usability and organization of the application. I would appreciate your thoughts and guidance on how we can address these concerns.

Thank you for your attention to these matters.

Error

Warning: mime_content_type(aarak traders OnlineCompanyRegistrationDetails-1.pdf): Failed to open stream: No such file or directory in C:\laragon\www\espofileapi\index.php on line 70

Warning: filesize(): stat failed for aarak traders OnlineCompanyRegistrationDetails-1.pdf in C:\laragon\www\espofileapi\index.php on line 71

Warning: file_get_contents(aarak traders OnlineCompanyRegistrationDetails-1.pdf): Failed to open stream: No such file or directory in C:\laragon\www\espofileapi\index.php on line 72