Automatic import from a file

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Vok
    Member
    • Jul 2020
    • 63

    Automatic import from a file

    Good afternoon, dear participants of the forum. I need to configure automatic import of contacts from a file. I followed this instruction :


    But there were no changes to ESPOCRM. Please help me set up automatic import from a file .
  • esforim
    Active Community Member
    • Jan 2020
    • 2204

    #2
    Maybe you want to paste your code and someone might be able to spot the mistake in the code.

    This of course assume you didn't fail these task (1) enable CRON is running and (2) file is in correct location.

    Comment


    • Vok
      Vok commented
      Editing a comment
      Thank you very much for your response. I understand how to connect CRON. I don't understand how to set up automatic import from a file. I would be very grateful if you would give me an example of the code and where to insert it. I'm a newbie, so I'm sorry for the stupid questions.
  • esforim
    Active Community Member
    • Jan 2020
    • 2204

    #3
    Hi Vok, unfortunately that is outside of my skill level. You said, "I followed this instruction" so I thought you have a working (draft) code that does the importing. All I'm trying to say is that if you copy/paste your code, someone might be able to see a mistake or provide tips/hints.

    Comment

    • item
      Active Community Member
      • Mar 2017
      • 1476

      #4
      Hello,
      "automatic import of contacts from a file"

      what kind of file is ? where is the file ? how is data in ?

      i do somethink like this..
      a first custom job :
      check if file exist on ftp server every hours
      if file exist .. download file and put to a custom folder

      a second custom job :
      check if file exist in custom folder.
      if file exist ... read file (csv) .. and "import"

      so i do the import in 2 jobs
      every month .. more than 150000 record are created so.

      You need, check duplicate and so ..
      If you could give the project a star on GitHub. EspoCrm believe our work truly deserves more recognition. Thanks.​

      Comment

      • Vok
        Member
        • Jul 2020
        • 63

        #5
        Originally posted by item
        Hello,
        "automatic import of contacts from a file"

        what kind of file is ? where is the file ? how is data in ?

        i do somethink like this..
        a first custom job :
        check if file exist on ftp server every hours
        if file exist .. download file and put to a custom folder

        a second custom job :
        check if file exist in custom folder.
        if file exist ... read file (csv) .. and "import"

        so i do the import in 2 jobs
        every month .. more than 150000 record are created so.

        You need, check duplicate and so ..
        Thank you very much for your response. You've written all the right things for me. But I can't figure out what to write after "public function run ()." in YourJobName.php

        How to specify that you need to import this particular file . I would be very grateful if you could share a sample or help me write the code

        Comment

        • Vok
          Member
          • Jul 2020
          • 63

          #6
          Here is my code in the task file :
          <?php

          namespace Espo\Custom\Jobs;

          class YourJobName extends \Espo\Core\Jobs\Base
          {

          public function run()
          {
          // all the logic needs to be defined in the method run
          $job = $this->getEntityManager()->getEntity('Job');
          $job->set(array(
          'serviceName' => $serviceName,
          'methodName' => $methodName,
          'data' => array('someVar' => 'someValue'),
          'executeTime' => date('Y-m-d H:i:s'),
          ));
          $this->getEntityManager()->saveEntity($job);
          }
          }

          But this code is created when I create a manual import :
          {
          "entityType": "Opportunity",
          "params": {
          "headerRow": true,
          "delimiter": ";",
          "textQualifier": """,
          "dateFormat": "YYYY-MM-DD",
          "timeFormat": "HH:mm",
          "timezone": "UTC",
          "personNameFormat": "f l",
          "decimalMark": ".",
          "currency": "RUB",
          "defaultValues": {},
          "action": "createAndUpdate",
          "skipDuplicateChecking": false,
          "idleMode": false,
          "silentMode": true,
          "updateBy": [
          8
          ]
          },
          "attachmentId": "5f2d1c98eccd50af8",
          "importAttributeList": [
          "dataT",
          "accountId",
          "accountName",
          "name",
          "kol",
          "amountCurrency",
          "amount",
          "stage",
          "accountFiscalNumber"
          ],
          "importId": "5f2d1c992e125ab06",
          "userId": "1"
          }

          Comment

          • telecastg
            Active Community Member
            • Jun 2018
            • 907

            #7
            We don't use Jobs in our application, but perhaps looking at a built in job script can help https://github.com/espocrm/espocrm/b...ifications.php

            Comment

            • item
              Active Community Member
              • Mar 2017
              • 1476

              #8
              Hello,

              here my sample .. i have don't write all because i don't know your file :


              PHP Code:
              <?php
              
              namespace Espo\Custom\Jobs;
              use Espo\ORM\Entity;
              
              class CareJobs extends \Espo\Core\Jobs\Base
              {
              public function run()
              {
              ini_set('memory_limit','16M');
              $fileName = 'data/xxxx/care.csv';
              if(!file_exists($fileName) || !is_readable($fileName))
              {
              $GLOBALS['log']->info( "CareJobs Nothing to process");
              return false;
              }
              $GLOBALS['log']->warning( "Begin Jobs CareJobs");
              if (($handle = fopen( $fileName, "r")) !== false) {
              $entityManager = $this->getEntityManager();
              $time_start = microtime(true);
              
              
              $header = fgetcsv($handle);
              while (($data = fgetcsv($handle, 2000, ",")) !== false) {
              $practriceArray = explode("_", $data[0] );
              $practrice = $practriceArray[1];
              ...
              this jobs import more than 150000 record each month
              as you can see.. another jobs check ftp for file.csv...if file exist download and put in folder of espocrm
              If you could give the project a star on GitHub. EspoCrm believe our work truly deserves more recognition. Thanks.​

              Comment

              • Vok
                Member
                • Jul 2020
                • 63

                #9
                Originally posted by item
                Hello,

                here my sample .. i have don't write all because i don't know your file :


                PHP Code:
                <?php
                
                namespace Espo\Custom\Jobs;
                use Espo\ORM\Entity;
                
                class CareJobs extends \Espo\Core\Jobs\Base
                {
                public function run()
                {
                ini_set('memory_limit','16M');
                $fileName = 'data/xxxx/care.csv';
                if(!file_exists($fileName) || !is_readable($fileName))
                {
                $GLOBALS['log']->info( "CareJobs Nothing to process");
                return false;
                }
                $GLOBALS['log']->warning( "Begin Jobs CareJobs");
                if (($handle = fopen( $fileName, "r")) !== false) {
                $entityManager = $this->getEntityManager();
                $time_start = microtime(true);
                
                
                $header = fgetcsv($handle);
                while (($data = fgetcsv($handle, 2000, ",")) !== false) {
                $practriceArray = explode("_", $data[0] );
                $practrice = $practriceArray[1];
                ...
                this jobs import more than 150000 record each month
                as you can see.. another jobs check ftp for file.csv...if file exist download and put in folder of espocrm
                Thank you so much for taking an interest in my question. I am very grateful to you!!!

                Comment

                • item
                  Active Community Member
                  • Mar 2017
                  • 1476

                  #10
                  Hello,

                  this jobs download the csv file.. put in folder .. and create the other jobs who inject csv to espoCRM

                  PHP Code:
                  <?php
                  
                  namespace Espo\Custom\Jobs;
                  use Espo\ORM\Entity;
                  use phpseclib\Net\SFTP;
                  
                  class SftpJobs extends \Espo\Core\Jobs\Base
                  {
                  
                  public function run()
                  {
                  $GLOBALS['log']->warning( "Start SftpJobs : " );
                  
                  $path = "/home/path/CSV/";
                  $fileDate = date('Y-n', strtotime(date('Y-m')." -1 month"));
                  $fileName = "xxx-" .$fileDate .".csv";
                  
                  $file = $path .$fileName;
                  
                  $sftp = new SFTP('10.10.10.10', 2121 );
                  $sftp_login = $sftp->login('login', 'password');
                  if($sftp_login) {
                  
                  if( $sftp->file_exists( $file)) {
                  $GLOBALS['log']->warning( 'SftpJobs : File Exist' );
                  $sftp->get($file, 'data/path/care.csv');
                  $sftp->rename($file, '/home/path/CSV/Backup/' .$fileName);
                  $GLOBALS['log']->warning( 'SftpJobs : File Downloaded & Moved To Backup : ' .$fileName );
                  $entityManager = $this->getEntityManager();
                  $job = $entityManager->getEntity('Job');
                  $job->set([
                  'name' => 'Care Jobs',
                  'status' => 'Pending',
                  'serviceName' => 'CareJobs',
                  'methodName' => 'run',
                  'data' => NULL, // array('' => ''),
                  'executeTime' => date('Y-m-d H:i:s',strtotime(date("Y-m-d H:i:s")." +5 minutes")),
                  'scheduledJobId' => '5c7e592209b492e20',
                  ]);
                  $entityManager->saveEntity($job);
                  }else{
                  $GLOBALS['log']->warning( 'SftpJobs : File Dont Exist');
                  }
                  }else{
                  $GLOBALS['log']->warning( 'SftpJobs : Cannot login into your server !');
                  }
                  
                  $GLOBALS['log']->warning( "End SftpJobs : " );
                  return true;
                  
                  }
                  }
                  If you could give the project a star on GitHub. EspoCrm believe our work truly deserves more recognition. Thanks.​

                  Comment

                  • Vok
                    Member
                    • Jul 2020
                    • 63

                    #11
                    Originally posted by item
                    Hello,

                    this jobs download the csv file.. put in folder .. and create the other jobs who inject csv to espoCRM

                    PHP Code:
                    <?php
                    
                    namespace Espo\Custom\Jobs;
                    use Espo\ORM\Entity;
                    use phpseclib\Net\SFTP;
                    
                    class SftpJobs extends \Espo\Core\Jobs\Base
                    {
                    
                    public function run()
                    {
                    $GLOBALS['log']->warning( "Start SftpJobs : " );
                    
                    $path = "/home/path/CSV/";
                    $fileDate = date('Y-n', strtotime(date('Y-m')." -1 month"));
                    $fileName = "xxx-" .$fileDate .".csv";
                    
                    $file = $path .$fileName;
                    
                    $sftp = new SFTP('10.10.10.10', 2121 );
                    $sftp_login = $sftp->login('login', 'password');
                    if($sftp_login) {
                    
                    if( $sftp->file_exists( $file)) {
                    $GLOBALS['log']->warning( 'SftpJobs : File Exist' );
                    $sftp->get($file, 'data/path/care.csv');
                    $sftp->rename($file, '/home/path/CSV/Backup/' .$fileName);
                    $GLOBALS['log']->warning( 'SftpJobs : File Downloaded & Moved To Backup : ' .$fileName );
                    $entityManager = $this->getEntityManager();
                    $job = $entityManager->getEntity('Job');
                    $job->set([
                    'name' => 'Care Jobs',
                    'status' => 'Pending',
                    'serviceName' => 'CareJobs',
                    'methodName' => 'run',
                    'data' => NULL, // array('' => ''),
                    'executeTime' => date('Y-m-d H:i:s',strtotime(date("Y-m-d H:i:s")." +5 minutes")),
                    'scheduledJobId' => '5c7e592209b492e20',
                    ]);
                    $entityManager->saveEntity($job);
                    }else{
                    $GLOBALS['log']->warning( 'SftpJobs : File Dont Exist');
                    }
                    }else{
                    $GLOBALS['log']->warning( 'SftpJobs : Cannot login into your server !');
                    }
                    
                    $GLOBALS['log']->warning( "End SftpJobs : " );
                    return true;
                    
                    }
                    }
                    Thanks a lot for your code and interest in my problem !! There is one last question left. Where do I need to indicate that the column from the CSV file corresponds to the column in ESPOCRM /
                    Example:
                    in the CSV file, the third column is the client's name. It is necessary that when importing, ESPOCRM understands that the third column is clients.

                    "importAttributeList": [
                    "dataT",
                    "accountId",
                    "accountName",
                    "name",
                    "kol",
                    "amountCurrency",
                    "amount",
                    "stage",
                    "accountFiscalNumber"
                    ],

                    Comment

                    • item
                      Active Community Member
                      • Mar 2017
                      • 1476

                      #12
                      Hello,

                      i use a csv from another provider .. who put in ftp ... i have give my job who check existence of the file .. if exist.. download to espocrm folder.. then make a new job

                      the job is just read csv file .. and make the "inject"..
                      yes i need to match first column to .. .second column to ...

                      all depend how are you csv

                      PHP Code:
                      $header = fgetcsv($handle);
                      while (($data = fgetcsv($handle, 2000, ",")) !== false) {
                      //$data = mb_convert_encoding($data, 'ISO-8859-1','ISO-8859-1');
                      $practriceArray = explode("_", $data[0] );
                      $practrice = $practriceArray[1];
                      
                      $inamiTo = substr($inamiNumberTo, 0, 8);
                      // ACCOUNT
                      $account = $entityManager->getRepository('Account')->where([ 'practrice' => $practrice ])->findOne();
                      if (!$account){
                      $account = $entityManager->getEntity('Account');
                      $account->set([
                      'name' => $practrice,
                      'practrice' => $practrice,
                      'teamsIds' => $teamsIds,
                      'assignedUserId' => '1'
                      ]);
                      $entityManager->saveEntity($account);
                      }//else{
                      // foreach ($account->get('teams') as $team) {
                      // $teamsIds[] = $team->id;
                      // }
                      //}
                      // CONTACT
                      $contact = $entityManager->getRepository('Contact')->where([ 'inami' => $inami ])->findOne();
                      if (!$contact){
                      $contact = $entityManager->getEntity('Contact');
                      $contact->set([
                      'firstName' => ucwords(mb_strtolower($data[16])),
                      'lastName' => mb_strtoupper( $data[15]) ,
                      'inamiNumber' => $inamiNumber,
                      'practrice' => $practrice,
                      'dateFirstCare' => $dt->format('Y-m-d'),
                      'teamsIds' => $teamsIds,
                      'assignedUserId' => '1'
                      ]);
                      $entityManager->saveEntity($contact);
                      }
                      
                      // PATIENT
                      $patient = $entityManager->getRepository('Patient')->where([ 'nationalNumber' => $nationalNumber ])->findOne();
                      if (!$patient){
                      $patient = $entityManager->getEntity('Patient');
                      $patient->set([
                      'firstName' => ucwords(mb_strtolower($data[2] )),
                      'lastName' => mb_strtoupper($data[1] ),
                      'addressStreet' => $addressStreet,
                      'addressPostalCode' => $addressPostalCode,
                      'addressCity' => $addressCity,
                      'nationalNumber' => $nationalNumber,
                      'cg1' => $cg1,
                      'cg2' => $cg2,
                      'patHealth' => $patHealth,
                      'score' => $score,
                      'dateFirstCare' => $dt->format('Y-m-d'),
                      'teamsIds' => $teamsIds,
                      'assignedUserId' => '1'
                      ]);
                      $entityManager->saveEntity($patient);
                      }
                      // CARE
                      $care = $entityManager->getEntity('Care');
                      $care->set(array(
                      'name' => $patient->get('name'),
                      'practrice' => $practrice,
                      'dateTime' => $dateTime,
                      ));
                      $entityManager->saveEntity($care);
                      
                      $convention = $entityManager->getRepository('Convention')->where([ 'name' => $year .$month ])->findOne();
                      if (!$convention){
                      $convention = $entityManager->getEntity('Convention');
                      $convention->set([
                      'name' => $year .$month,
                      'dateStart' => date('Y-m-01', strtotime($care->get('dateTime'))),
                      'dateEnd' => date('Y-m-t', strtotime($care->get('dateTime'))),
                      'teamsIds' => $teamsIds,
                      'assignedUserId' => '1',
                      ]);
                      $entityManager->saveEntity($convention);
                      }
                      
                      if (!$entityManager->getRepository('Convention')->isRelated($convention, 'contacts', $contact)){
                      $entityManager->getRepository('Convention')->relate($convention, 'contacts', $contact);
                      }
                      
                      
                      }
                      fclose($handle);
                      } 
                      
                      If you could give the project a star on GitHub. EspoCrm believe our work truly deserves more recognition. Thanks.​

                      Comment

                      • Vok
                        Member
                        • Jul 2020
                        • 63

                        #13
                        Thanks again for the code! There will be something to keep yourself busy on the day

                        Comment

                        Working...