Request for Mass Import of 200 Documents into Documents Entity

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • sapyantuk
    Senior Member
    • Oct 2020
    • 265

    Request for Mass Import of 200 Documents into Documents Entity



    Dear Developers,

    I hope this message finds you well. I am reaching out to discuss an important task that requires your expertise. Currently, I am faced with the challenge of uploading a significant number of documents, specifically over 200, into the Documents Entity.

    As you can imagine, manually uploading each document one by one would be a time-consuming and laborious process. Hence, I am eager to explore alternative methods that could streamline this task and improve efficiency. In this regard, I am wondering if there is a way to perform a mass import of these 200 documents.
  • item
    Active Community Member
    • Mar 2017
    • 1476

    #2
    Hi,
    need développer skill for me and script that loop over a directory and post attachment and post document


    If you could give the project a star on GitHub. EspoCrm believe our work truly deserves more recognition. Thanks.​

    Comment

    • rabii
      Active Community Member
      • Jun 2016
      • 1250

      #3
      hey sapyantuk

      Just in case you still need a tool to help you upload the files to the server, please find attached a small php app which will allow you to do that, the way it works if that you unzip the file and just drop it to a php web server and access the app folder, when you access the web folder you should be able to modify index.php (add your crm domain and your api key) once that is done, then you can access the application and provide the full path to the directory where the files exist, and press upload the files. the application will scan the files directory and get rid of any . or .. files and then will loop through the files and send a post request to create an attachment and get returned id and after that send another post request to create a document and related the attachment to the document.

      i would advise you to try this out with one or two files and once you are happy with the result then do it for all 200 files.

      here is a video that shows how it works.


      I hope this helps.
      Attached Files
      Rabii
      Web Dev

      Comment

      • sapyantuk
        Senior Member
        • Oct 2020
        • 265

        #4
        Thank you, rabii , for your response.

        I have files on my system organized within folders, and I would like to upload these files while preserving the folder structure. For example, I have a file named "abc.docx" located inside a folder called "Abc". Is there a way to achieve uploading these files with the same folder structure intact?

        Comment


        • rabii
          rabii commented
          Editing a comment
          It is possible to apply such logic, it will just add necessity to scan subfolder and create them in the process, let me see if i can update the code to add this functionality for you.
      • esforim
        Active Community Member
        • Jan 2020
        • 2204

        #5
        Wow I thought this was something hard to use but looking at the video it so easy!

        And lol these filler time in video:
        Click image for larger version

Name:	image.png
Views:	288
Size:	36.7 KB
ID:	92654

        Comment


        • rabii
          rabii commented
          Editing a comment
          i was doing the video while working on something else so i was in rush and have few missed typing ??

          Eevrything is possible with Espocrm mate, i am just diving into some cool staff you can do with the system.
      • sapyantuk
        Senior Member
        • Oct 2020
        • 265

        #6
        rabii any update on uploading files with the same folder structure ?

        Comment

        • rabii
          Active Community Member
          • Jun 2016
          • 1250

          #7
          sapyantuk

          Sorry i was busy with some urgent work. i have modified the code to loop through a directory and get the subdirectories and then create documentfolder record via post request, after that loop through each subdirectory and create attachment, document and link them - also document will be linked with the correct directory. i have not had time to properly display any errors but just echo them into the screen if there are any errors.

          Make sure your api user has the permission to create DocumentFolder and Document in their role. Just provide the full path to the directory (main directory) and press upload files button. it might take some times if the folders and files are big in size.

          Please test this out with few files before applying this on the 200 pdf files. also if you have any images / audio etc make sure the file field in documents accept those extensions.

          find attached modified version.

          Thanks
          Attached Files
          Rabii
          Web Dev

          Comment


          • esforim
            esforim commented
            Editing a comment
            Do you have a git or somewhere I can automatic stalk you
        • sapyantuk
          Senior Member
          • Oct 2020
          • 265

          #8
          rabii


          Dear Rabii,

          I hope this message finds you well. I wanted to inform you that I have successfully uploaded the file you provided. Additionally, I have made the necessary changes by replacing the ESPo CRM URL and the API code in the "index.php" file.

          However, I have encountered an issue when entering the local system's full path. It consistently displays an error message stating "Invalid path." Furthermore, on some occasions, I am confronted with an HTTP ERROR 500.

          I would greatly appreciate your assistance in resolving this matter. Please let me know if you require any further information or if there are any specific steps I should take to address this issue.

          Thank you for your attention to this matter.

          Comment


          • rabii
            rabii commented
            Editing a comment
            Hey
            "Invalid path." means that the path that you provided is not correct it should be something like (C:\main-folder\documents-folder) on mac you could use the pwd on the main folder to get the full path.

            HTTP ERROR 500 Means some resources / records were not created (this could be cause by different issues e.g the file extension is not permitted on the file on your entity).

            do you check the crm to see if all necessary folders / documents and attachments are created?

          • sapyantuk
            sapyantuk commented
            Editing a comment
            The path I entered is as follows: C:\Users\Auditpcmain\Desktop\folder. However, upon entering this path, I received an error message stating "Invalid path."

            To further investigate this issue, I have checked the CRM system, and it appears that nothing has been created or associated with the provided path. It seems that there might be a problem with the path itself or with the system's configuration.

            I kindly request your assistance in resolving this matter. Please let me know if you require any additional information or if there are any specific steps I should take to troubleshoot the path issue.
        • sapyantuk
          Senior Member
          • Oct 2020
          • 265

          #9
          rabii
          The path I entered is as follows: C:\Users\Auditpcmain\Desktop\folder. However, upon entering this path, I received an error message stating "Invalid path."

          To further investigate this issue, I have checked the CRM system, and it appears that nothing has been created or associated with the provided path. It seems that there might be a problem with the path itself or with the system's configuration.

          I kindly request your assistance in resolving this matter. Please let me know if you require any additional information or if there are any specific steps I should take to troubleshoot the path issue.

          Comment


          • rabii
            rabii commented
            Editing a comment
            does it say (Invalid directory path) ?

          • sapyantuk
            sapyantuk commented
            Editing a comment
            Yes. Sometimes , i couldnt reproduce all time . sometime it says Invalid directory path and most of the time HTTP ERROR 500
            If you are available please join teamviewer for 2 min
            822 816 264
            Last edited by sapyantuk; 06-02-2023, 11:45 AM.

          • rabii
            rabii commented
            Editing a comment
            The error 500 is the returned error because no record was being able to be created. i have noticed that you have a wrong path if you are on windows it should (C:/Users/Auditpcmain/Desktop/folder) otherwise the it will trigger an error because php will not be able to read the path correctly. use the forward slash.

            This should be the correct path C:/Users/Auditpcmain/Desktop/folder
        • sapyantuk
          Senior Member
          • Oct 2020
          • 265

          #10
          rabii i tried
            • C:/Users/Auditpcmain/Desktop/folder
          ​still same issue

          Comment

          • rabii
            Active Community Member
            • Jun 2016
            • 1250

            #11
            as mentioned before please try to test this on one folder and one file because you proceed with the 200 files as this could cause a problem of memory limit and therefore it will cause issues.
            Rabii
            Web Dev

            Comment


            • sapyantuk
              sapyantuk commented
              Editing a comment
              7q92r8us


              this is the password
          • rabii
            Active Community Member
            • Jun 2016
            • 1250

            #12
            what teamview password
            Rabii
            Web Dev

            Comment

            • sapyantuk
              Senior Member
              • Oct 2020
              • 265

              #13
              7q92r8us


              password

              Comment

              • sapyantuk
                Senior Member
                • Oct 2020
                • 265

                #14
                rabii


                I have installed a local environment called XAMPP and attempted to use it, but I am still encountering an error. I have verified that the cURL extension is installed, yet the error persists.​

                Click image for larger version

Name:	image.png
Views:	255
Size:	37.4 KB
ID:	93250​​

                Comment

                • sapyantuk
                  Senior Member
                  • Oct 2020
                  • 265

                  #15
                  rabii

                  I hope this message finds you well. I wanted to inform you that I have successfully installed the local environment that you suggested, and as a result, the application has started working. However, I encountered an error that I'd like to discuss.

                  The issue arises when we upload a document through the portal. Although records are being created, the problem lies in opening or downloading the file itself. It appears that the size of the file is consistently shown as 0 KB, indicating that there is no content within it.

                  Upon conducting a thorough analysis of the issue, I discovered the root cause. The files are being scanned from the input path as expected, but the code seems to be searching for these files within the home directory where the application is hosted. This discrepancy in the search locations is likely the reason behind the empty files.

                  Additionally, I wanted to bring to your attention another issue that I've come across. When we create or send files through the application, it seems that the files are not being posted in the same folder that already exists in Espo.

                  To illustrate, let's say there is a folder in Espo called "Account." When I have some files on my local computer and attempt to upload them, the API creates a new folder named "Account" instead of posting the files in the existing directory in Espo.

                  I was wondering if it would be possible to implement a check for the folder name. If the folder name already exists, we can then post the files in the existing directory in Espo. This would allow us to upload the documents in our desired structure by leveraging the same folder structure that exists in Espo.

                  Combining the resolution of the file size issue and the implementation of the folder check feature would greatly improve the usability and organization of the application. I would appreciate your thoughts and guidance on how we can address these concerns.

                  Thank you for your attention to these matters.

                  Error

                  Warning: mime_content_type(aarak traders OnlineCompanyRegistrationDetails-1.pdf): Failed to open stream: No such file or directory in C:\laragon\www\espofileapi\index.php on line 70

                  Warning: filesize(): stat failed for aarak traders OnlineCompanyRegistrationDetails-1.pdf in C:\laragon\www\espofileapi\index.php on line 71

                  Warning: file_get_contents(aarak traders OnlineCompanyRegistrationDetails-1.pdf): Failed to open stream: No such file or directory in C:\laragon\www\espofileapi\index.php on line 72

                  Comment

                  Working...