Locking of requests during Email downloads

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • david.strejc
    Member
    • Jul 2016
    • 38

    Locking of requests during Email downloads

    We are experiencing aproximately 25s delay in loading of Account entity when there is ongoing Email downloads.

    Is this behaviour normal or is there some error in our setup of Espo?

    This is little bit annoying for our team.

    Thanks for reply.
  • Maximus
    Senior Member
    • Nov 2018
    • 2731

    #2
    Hello David.
    It is hard to say what exactly the cause of this. The loading time depends on many factors like:
    - Server Performance.
    - Allocated server's RAM memory.
    - DB version.
    - How many accounts do you have in the Account table in the DB?
    - How many columns do you have in the Account entity List view?
    - Whether the show total Accounts number on the list view is enabled?
    - Search filters.
    - Indexes.
    - etc.

    Comment

    • david.strejc
      Member
      • Jul 2016
      • 38

      #3
      Originally posted by Maximus
      Hello David.
      It is hard to say what exactly the cause of this. The loading time depends on many factors like:
      - Server Performance.
      - Allocated server's RAM memory.
      - DB version.
      - How many accounts do you have in the Account table in the DB?
      - How many columns do you have in the Account entity List view?
      - Whether the show total Accounts number on the list view is enabled?
      - Search filters.
      - Indexes.
      - etc.
      Many thanks for reply Maximus.

      I am long time Unix admin and I little bit know what I am doing.
      So for info:

      4G dedicated RAM (server has 64GB)
      MariaDB 10.3
      approx 2200 accounts
      39 columns in DB for table account

      PHP Code:
      MariaDB [project_alis_tech_com]> SHOW INDEX FROM account;
      +---------+------------+----------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
      | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
      +---------+------------+----------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
      | account | 0 | PRIMARY | 1 | id | A | 2516 | NULL | NULL | | BTREE | | |
      | account | 0 | UNIQ_CREATED_AT_ID | 1 | created_at | A | 629 | NULL | NULL | YES | BTREE | | |
      | account | 0 | UNIQ_CREATED_AT_ID | 2 | id | A | 2516 | NULL | NULL | | BTREE | | |
      | account | 1 | IDX_CAMPAIGN_ID | 1 | campaign_id | A | 2 | NULL | NULL | YES | BTREE | | |
      | account | 1 | IDX_CREATED_BY_ID | 1 | created_by_id | A | 20 | NULL | NULL | YES | BTREE | | |
      | account | 1 | IDX_MODIFIED_BY_ID | 1 | modified_by_id | A | 22 | NULL | NULL | YES | BTREE | | |
      | account | 1 | IDX_ASSIGNED_USER_ID | 1 | assigned_user_id | A | 22 | NULL | NULL | YES | BTREE | | |
      | account | 1 | IDX_CREATED_AT | 1 | created_at | A | 629 | NULL | NULL | YES | BTREE | | |
      | account | 1 | IDX_CREATED_AT | 2 | deleted | A | 629 | NULL | NULL | YES | BTREE | | |
      | account | 1 | IDX_NAME | 1 | name | A | 2516 | NULL | NULL | YES | BTREE | | |
      | account | 1 | IDX_NAME | 2 | deleted | A | 2516 | NULL | NULL | YES | BTREE | | |
      | account | 1 | IDX_ASSIGNED_USER | 1 | assigned_user_id | A | 22 | NULL | NULL | YES | BTREE | | |
      | account | 1 | IDX_ASSIGNED_USER | 2 | deleted | A | 24 | NULL | NULL | YES | BTREE | | |
      +---------+------------+----------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
      13 rows in set (0.001 sec) 
      
      But system is behaving REALLY WELL - with one exception and that is:

      * when there is running job for downloading email attachments whole system is locked for duration of email downloads
      * this is of course random as there is no way to tell when system will download emails

      Standard request:

      curl -s -w "@curl-format.txt" '/api/v1/Account?maxSize=100&offset=0&orderBy=createdAt&ord er=desc'

      takes 0.7s

      But when there is cron downloading emails it always takes approx 25s - depends on size of email.

      We went through code and it seems that system is locked by "locker" class in Entity manager for some reason even when there is ongoing download of email attachments (which is strange because this has nothing to do with DB right?)

      Comment

      • esforim
        Active Community Member
        • Jan 2020
        • 2204

        #4
        Whenever I see MariaDB nowadays I just think of the recent issue of Collation, have you looked at this slowdown issue with MariaDB yet David? I wonder if you were affected as well: https://forum.espocrm.com/forum/anno...7358#post67358

        Second question is, has this always been happening ever since you install your Espo or it only recently during an update? It only becoming more slower and you starting to notice it now.

        Looking at real-world, has anything change recently that may increase in usage? E.g. You install a new software or you just hire 100 more staff.

        Whatever is the issue is, it probably out of my skill level, but hopefully these idea can help you somehow, if it not already been done through your experience or checklist.

        Comment

        • david.strejc
          Member
          • Jul 2016
          • 38

          #5
          Originally posted by espcrm
          Whenever I see MariaDB nowadays I just think of the recent issue of Collation, have you looked at this slowdown issue with MariaDB yet David? I wonder if you were affected as well: https://forum.espocrm.com/forum/anno...7358#post67358

          Second question is, has this always been happening ever since you install your Espo or it only recently during an update? It only becoming more slower and you starting to notice it now.

          Looking at real-world, has anything change recently that may increase in usage? E.g. You install a new software or you just hire 100 more staff.

          Whatever is the issue is, it probably out of my skill level, but hopefully these idea can help you somehow, if it not already been done through your experience or checklist.
          Many thanks - this particular issue was caused by "currently unknown" error in select button for related entity which is for unknown reason at Quotes entity selecting whole set of data (including every join - so this means 5 tables joined" resulting in this select


          PHP Code:
          SELECT account.id AS `id`, account.name AS `name`, account.deleted AS `deleted`, account.website AS `website`, emailAddresses.name AS `emailAddress`, phoneNumbers.name AS `phoneNumber`, account.type AS `type`, account.industry AS `industry`, account.sic_code AS `sicCode`, LEFT(account.billing_address_street, 5000) AS `billingAddressStreet`, account.billing_address_city AS `billingAddressCity`, account.billing_address_state AS `billingAddressState`, account.billing_address_country AS `billingAddressCountry`, account.billing_address_postal_code AS `billingAddressPostalCode`, LEFT(account.shipping_address_street, 5000) AS `shippingAddressStreet`, account.shipping_address_city AS `shippingAddressCity`, account.shipping_address_state AS `shippingAddressState`, account.shipping_address_country AS `shippingAddressCountry`, account.shipping_address_postal_code AS `shippingAddressPostalCode`, LEFT(account.description, 5000) AS `description`, account.created_at AS `createdAt`, account.modified_at AS `modifiedAt`, LEFT(account.dic, 5000) AS `dic`, account.issupplier AS `issupplier`, account.iscustomer AS `iscustomer`, account.activeclientwp AS `activeclientwp`, account.activeclientcrm AS `activeclientcrm`, account.change_ico AS `changeIco`, account.spolecnosti AS `spolecnosti`, account.partner AS `partner`, LEFT(account.firstname, 5000) AS `firstname`, LEFT(account.lastname, 5000) AS `lastname`, LEFT(account.idph, 5000) AS `idph`, account.category AS `category`, account.language AS `language`, account.enduser AS `enduser`, emailAddresses.opt_out AS `emailAddressIsOptedOut`, phoneNumbers.opt_out AS `phoneNumberIsOptedOut`, account.campaign_id AS `campaignId`, campaign.name AS `campaignName`, account.created_by_id AS `createdById`, TRIM(CONCAT(IFNULL(createdBy.first_name, ''), ' ', IFNULL(createdBy.last_name, ''))) AS `createdByName`, account.modified_by_id AS `modifiedById`, TRIM(CONCAT(IFNULL(modifiedBy.first_name, ''), ' ', IFNULL(modifiedBy.last_name, ''))) AS `modifiedByName`, account.assigned_user_id AS `assignedUserId`, TRIM(CONCAT(IFNULL(assignedUser.first_name, ''), ' ', IFNULL(assignedUser.last_name, ''))) AS `assignedUserName`, originalLead.id AS `originalLeadId`, TRIM(CONCAT(IFNULL(originalLead.first_name, ''), ' ', IFNULL(originalLead.last_name, ''))) AS `originalLeadName`, account.ploha_id AS `plohaId`, ploha.name AS `plohaName` FROM `account` LEFT JOIN `attachment` AS `ploha` ON account.ploha_id = ploha.id LEFT JOIN `lead` AS `originalLead` ON account.id = originalLead.created_account_id AND originalLead.deleted = 0 LEFT JOIN `campaign` AS `campaign` ON account.campaign_id = campaign.id LEFT JOIN `user` AS `assignedUser` ON account.assigned_user_id = assignedUser.id LEFT JOIN `user` AS `modifiedBy` ON account.modified_by_id = modifiedBy.id LEFT JOIN `user` AS `createdBy` ON account.created_by_id = createdBy.id LEFT JOIN `entity_email_address` AS `emailAddressesMiddle` ON account.id = emailAddressesMiddle.entity_id AND emailAddressesMiddle.deleted = 0 AND emailAddressesMiddle.primary = 1 AND emailAddressesMiddle.entity_type = 'Account' LEFT JOIN `email_address` AS `emailAddresses` ON emailAddresses.id = emailAddressesMiddle.email_address_id AND emailAddresses.deleted = 0 LEFT JOIN `entity_phone_number` AS `phoneNumbersMiddle` ON account.id = phoneNumbersMiddle.entity_id AND phoneNumbersMiddle.deleted = 0 AND phoneNumbersMiddle.primary = 1 AND phoneNumbersMiddle.entity_type = 'Account' LEFT JOIN `phone_number` AS `phoneNumbers` ON phoneNumbers.id = phoneNumbersMiddle.phone_number_id AND phoneNumbers.deleted = 0 WHERE account.deleted = 0 ORDER BY account.created_at DESC, account.id DESC LIMIT 0, 101; 
          
          Examining 123 000 000 lines. And this took 25s

          Comment

          • item
            Active Community Member
            • Mar 2017
            • 1476

            #6
            Hello,
            witch version espocrm ?
            witch field on layout ?

            Hi All, After upgrade (5.8.4->5.8.5->5.9.3) i found that in new version when you create new entity, primary an foreign keys are created with different collation that was before. Older one for keys was "utf8_unicode_ci", now it is utf8mb4_unicode_ci. It is a big problem because when you join fields with


            i have remove all your custom field and execute the sql on my computer : 30ms
            you have increase to 101 listview record ?

            can you remove from layout this field LEFT JOIN `attachment` AS `ploha` ON account.ploha_id = ploha.id
            i don't see ploha_id on index .. maybe you need run Yuri script.

            test on clone instance
            If you could give the project a star on GitHub. EspoCrm believe our work truly deserves more recognition. Thanks.​

            Comment

            Working...