EspoCRM 6.1.0 released

[emillod]

esforim i'm not sure why, but i think after upgrade loading of emails take more than 5 seconds. It worked faster before. I'll investigate that issue, maybe i find some solution.
I wonder if it's connected to upgrade or to amount of emails. But amount of emails is not big.

[esforim]

Hope to hear from you. Email is especially bad, but all my other entity search is very slow as well.

Not sure how I can investigate this further.

[emillod]

esforim we're planning about rebuilding indexes in email table. For now i have two production instaned on 6.1 and i see on one instance issue with loading email. On other i don't see any BIG issues, but there we have less data.

[murugappan]

I had similar problem when i upgraded from 5.9 to 6.0.1. The List for one the entities was extremely slow. For the entity, i I have a field that links to another entity. i.e, The List for GL Entity shows GL Number, Claim Number, Patient Name, Hospital Name and Status. The field i am talking about is the Hospital Name which a link to another entity called Hospitals. I decided to sacrifice the Hospital Name and the List became very fast (instantaneous). Now i am having a odd problem with upgrading to 6.1.2 (see my other post below). It appears that the rebuild fails.

[davide445]

davide445 to add grouping option, you should open settings of espo or your user preferences and scroll to section where you're managing menu. After that you need to click Add new item and in the popup you'll see button to add group tab.
Can you tell me more about attachments broken links? Do you have any errors in logs?

We needed to rollback Espo app to 2.0.9 using file backup, didnt touched the DB so far.
Next you can download the log of the installation (see the one at 7AM), seems there was many errors.

[TomV]

FAILED CLI upgrade from 6.0.10 to 6.1.0. Just as per the instructions ran

 php command.php upgrade

and received error in the log file:

ALERT: Rebuild database fault: PDOException: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'address_postal_code' at row 282 in /usr/share/nginx/html/vendor/doctrine/dbal/src/Driver/PDO/Connection.php:82 Stack

I just listed all my postal codes in both accounts and contacts, and was able to check. But they all seem fine...

[yuri]

Did you check deleted records? What length contact.address_postal_code column has?

[TomV]

In 'Administration >Entity Manage > Contact > Fields' The field 'addressPostalCode' had a length of 80, I changed to 1500 and re-run the upgrade sequence, but to no avail. Still same error.
yuri - any news on this? I suppose until we found the bug it might make sense to pull this upgrade from your repo?

[TomV]

I tried to run the upgrade again this morning and I can confirm this problem has been solved )))... We are now on 6.1.4 :-)
Super - thanks ))))

[emillod]

yuri i think there is an issue with layout settings. I've reproduce issue on demo.espocrm.com site, so it's not only on my instance.
I've tried to add new fields to small list view in invoice entity, i saved new layout without issues but i don't see any changes. I refreshed website with settings and everything is there but no changes.

Settings of small list: https://i.imgur.com/5qcVT21.png
Screenshot of relation panel after change: https://i.imgur.com/qDlco82.png

[yuri]

It's not related to the new release There is a separate layout listForAccounts. The layout manager is working fine.

[emillod]

yuri oh! I missed that! I'm sorry, you're right!

[azidevnom]

After this update our database tanked pretty bad. And I mean PRETTY BAD, all cores went to 100% usage from mysql (mariadb 10.3) and espo just became unusable.

[esforim]

I think your issue is related with mine. Some post here: https://forum.espocrm.com/forum/anno...6819#post66819

Checking my server usage it seem fine though (CPU, Memory, etc), but not sure how I can check the Database usage query and it speed. It would explain why my search are so slow.

Does your data/logs have any error such as PDO_exception? or HYP2000

[esforim]

I revert the file back to v6.0.9 but there does seem to be any changed in term of speed. It still slow. So it might not be a file issue is what I'm thinking.

[murugappan]

I have decided to stop with 6.0.10 upgrade until there is some clear answers to some of the issues. Refer to the post #21 which is my problem.

[mbrands]

After this update our database tanked pretty bad. And I mean PRETTY BAD, all cores went to 100% usage from mysql (mariadb 10.3) and espo just became unusable.

Same here, for me its the contacts (and lesser extent the accounts) entity that keeps everything busy at 100%.
I can't seem to find the reason either

[yuri]

Did you check whether DB indexes are intact?

[mbrands]

Could you direct me on how i could check that?

[esforim]

UPDATE: found it. After opening contact you can open Structure. Scroll to the bottom and you will have Indexes.


How did you open this screen? I want to check my database too but can't find it. I followed this "guide" here but can't find any "index" or "go" either.

https://stackoverflow.com/questions/36736451/how-to-do-indexing-of-database-tables-in-phpmyadmin

Using myphpadmin > structure > contact

[mbrands]

I have some older backups and the indexes are the same. I think murugappan might be on to something, but the biggest problem with the contacts list is that is linked to accounts, calls, etc etc. I rather not brake that functionality. Since the update there is a change, but i havnt found it yet.

[esforim]

I follow murugappan method of removing my Small List and List view to just show "Name" field, no difference in the speed.

Here are my "performance", all these previously take less one 1 second or 0.5 second to load and search, but now:

Searching for a contact name in Meeting field ~30 second
Opening the Small List view of Contact ~30 second as well
--> Searching for a contact in Small List view ~15 second.

Opening the #contact entity (List View), I click on the Search (blank input) for it to do a refresh. ~50 second.

Records wise there is approximately ~2,200 contact.

I did these test by doing Clear Cache and Rebuild each time, and I did it a few time get the average speed is like that.

I wonder (and hopefully) if it related to this bug:

https://github.com/espocrm/espocrm/issues/1909

If it is then hopefully problem is resolved after this patch.

[esforim]

Is this an "working indexes"? This is for Contact. Looking at the number it seem good (?) Cardinality = numbers of records?

The Null part is bit worrying since it say "No" on 2 row but seem normal when comparing to mbrands screenshot.

[esforim]

I did use 6.1.2 and all my search/view is slowed down. So I reverted back to 6.0.9 hoping it would fix this issue (it didn't).

I will give you idea an tried (https://forum.espocrm.com/forum/anno...7100#post67100 ), I will trim down the list view field and see how that behave.

[esforim]

Regarding Logs, there doesn't seem to be an error related to overall performance of the system. I'm still getting this error for my email but I don't think it related?

[2021-02-05 00:39:28] Espo.ERROR: CronManager: Failed job running, job [601c]. Error Details: Job CheckEmailAccounts 5ffe791: [HY000] SQLSTATE[HY000]: General error: 2006 MySQL server has gone away at /home/u874062233/domains/espocrm.com/public_html/crm/application/Espo/Modules/Crm/Jobs/CheckEmailAccounts.php:74 [] []
[2021-02-05 00:39:28] Espo.ERROR: Uncaught Exception PDOException: "SQLSTATE[HY000]: General error: 2006 MySQL server has gone away" at /home/user/domains/espocrm.com/public_html/crm/application/Espo/ORM/SqlExecutor.php line 72 {"exception":"[object] (PDOException(code: HY000): SQLSTATE[HY000]: General error: 2006 MySQL server has gone away at /home/user/domains/espocrm.com/public_html/crm/application/Espo/ORM/SqlExecutor.php:72)"} []

But I do feel like it may be related to this HY000 and PDOexception error, Hence I asked this question: https://forum.espocrm.com/forum/anno...6971#post66971

Please note I filtered out of private information, so don't question that (e.g. Espocrm.com or (Email ID/CRON job ID)

[murugappan]

Something is seriously wrong with the latest version 6.1.2. I tested the upgrade with by upgrading fresh version 5.9.3 and everything was fine. But I upgraded any of my existing instances they all failed. Initial problems seems to be coming from 6.0.8 when i first encountered terrible performance issue on List for one of the entities.

[yuri]

murugappan I created a separate topic https://forum.espocrm.com/forum/inst...data-truncated

[murugappan]

Maximus requested be to contact support to help upload the dB Dump but the support is only available for paid users. Do i have to be a paid user even if it is a release bug?

[yuri]

For those who have performance issue after upgrade. It happens only on MariaDB if you have collation for ID columns set to utf8. If you change the collation for all id columns to utf8mb4 it should fix the problem.

[murugappan]

Thank you so much for the update. Suggest, that the next release should include a separate item on database configuration requirements. That way we would have done this before the upgrade. Hope my suggestion is acceptable.

After further investigation, I found that many of the key tables had Utf8 collation for the id. This happens to espocrm instances that have been upgraded from older versions. The fresh installation of espocrm (latest) sets the collation to utf8mb4. Only problem of resetting this collation is that there is warning that id field may get garbled. Risky!

Yuri, unfortunately, when we create a new entity, espo creates the table with the collation for key fields as utf8 instead of utf8mb4 even after upgrading.

[item]

I can confirm too
speed increased when i changed in my instance.
but i think we need to change all field in database, createdAt, createdById, ...modidiedById... modifiedAt .. and so .. maybe a script will be usefull.
Regards

[esforim]

I'm a total newbie when it come to database, any guide and quick tip on how I should go about doing this? Don't want to have to panic if it went wrong. Planning to use myPHPAdmin for this, would this be the right tools?

[esforim]

I manage to find the "collation" after I open an entity such as "Contact > Structure", in one of the column it say Collation. currently for the row name: id it say

utf8_unicode_ci so I guess we need to click "Change" then in Collation find utf8mb4 and save? Any other quick way to do this for all entity or will there be a "hotfix" patch? Lastly since previously my Collation was utf8_unicode_ci then I need to change to to utf8mb4_unicode_ci right?

[yuri]

Important is that all ID fields (id, contact_id, ids in relation tables) must be of the same collation. utf8 (actually it is utf8mb3) is obsolete and we are phasing it out. Older versions of EspoCRM had utf8 by default because at that time is there was a problem to support mb4 on all database engines.

We recommend to convert all columns to utf8_unicode_ci. We also recommend not to use MariaDB 10.1 but higher.

What happened in the upgrade. After we updated Doctrine DBAL because of some bug it converted some id columns to mb4. So there were different collations of id and e.g. contact_id columns.

[murugappan]

Espcrm, if your instance was created prior to 5.9.4, the collation will continue to be the old setting. That is the problem after we migrate to 6.0.8 and MariaDB 10.1 or greater. The new version of crm creates the IDX with the correct collation.

[esforim]

I think I was on 5.4 when I first use EspoCRM so for sure the database is created before 5.9.4, whichever was the latest release after Jan 2020 (my account forum date) would be the first/second database creation.

I was previously using a host with MySQL but one of the update obsoleted the PHP version 5 and had to move host, the new host used MariaDB.

Just to confirm yuri, " We recommend to convert all columns to utf8_unicode_ci. We also recommend not to use MariaDB 10.1 but higher."

You mean utf8mb4_unicode_ci right? Especially for MariaDB server, only use utf8_unicode_ci if you have MySQL?

[mbrands]

esforim You can do it for the whole database at ones from phpMyAdmin. See the screenshot.
But as always: BACKUP first

[esforim]

Thank mbrands, just did that. Hopefully no issue is found.

But is doing this OK? For example, I'm just scare that not every collation should be utf8mb4_unicode_ci, some have to be different type or mix.

But my email is finally working again!

[mbrands]

As far as i am aware, uft8mb4 is a safe bet when comming from uft8. Glad you got your issues solved aswell!

[item]

Hello Yuri,

can you confirm i can execute this ?

$sql = "SHOW TABLES";
$sth = $pdo->prepare($sql);
$sth->execute();
$tableList= $sth->fetchAll();
foreach ($tableList as $table) {
$sql = 'ALTER TABLE ' .$table[0] .' CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;' ;
$pdo->prepare($sql);
$sth->execute();
} 

i have tested on 2 table.. no issue just one bug because in dateField there are bad date value

for test purpose, one table with 6500000 record with many relation... 14min on my computer for the sql above alter mytable

i think need to add 2 more sql like this in the foreach

alter table $table[0] add key(id) ;
OPTIMIZE TABLE $table[0];

[esforim]

I follow mbrands method using myPHPadmin, it took about 2 minutes for me before a message like this "Your executions has been completed" popup

[murugappan]

I would recommend the same as what mbrand has suggested. It would be safer and tested.

[esforim]

Hi item, hdijkema also have similar to you, he did some "Alter Table", I'm not sure if it relevant to you but you might want to see this issue here: https://github.com/espocrm/espocrm/issues/1904

[yuri]

The script for conversion to MB4 can be found here: https://forum.espocrm.com/forum/inst...1212#post61212

[esforim]

Hi guys just want to update fellow MariaDB people, this "hotfix" should (hopefully) automatic solve issue of MariaDB server:

https://github.com/espocrm/espocrm/issues/1921

---

Just upgrade to 6.1.3 and run into an issue though, anyone having same problem? Seem to be related to the patch of "opcache" https://github.com/espocrm/espocrm/issues/1909

Getting blank page at the moment, no console error but there is error in logs. My error log is as below, already tried to replace the Manager.php to a previous version

Espo.ERROR: Uncaught Exception TypeError: "Argument 1 passed to Espo\Core\Utils\File\Manager::__construct() must be of the type array or null,
object given, called in /home/user/domains/espo.com/public_html/crm/application/Espo/Core/Loaders/FileManager.php on line 48" at /home/user/domains
/espo.com/public_html/crm/application/Espo/Core/Utils/File/Manager.php line 58 {"exception":"[object] (TypeError(code: 0): Argument 1 passed to Espo\\Core
\\Utils\\File\\Manager::__construct() must be of the type array or null, object given, called in /home/user/domains/espo.com/public_html/crm/application
/Espo/Core/Loaders/FileManager.php on line 48 at /home/user/domains/espo.com/public_html/crm/application/Espo/Core/Utils/File/Manager.php:58)"} []

Line 48 refer to this code: protected $tmpDir = 'data/tmp';
and Line 58: public function __construct(?array $defaultPermissions = null)

I notice my error have this file "/application/Espo/Core/Loaders/FileManager.php" but when I look at Github I don't see this file: https://github.com/espocrm/espocrm/t...o/Core/Loaders


I finally manage to get the page to load and not be blank by replacing the Manager.php from version 6.0.9 but not got a even better error need to solve.

Espo.ERROR: Slim Application Error Type: Error Code: 0 Message: Call to undefined method Espo\ORM\QueryParams\Select::getRaw() File: /home/user/domains/crm.com/public_html/crm/application/Espo/ORM/QueryComposer/BaseQueryComposer.php Line: 248 Trace: #0 /home/user/domains/crm.com/public_html/crm/application/Espo/ORM/QueryComposer/BaseQueryComposer.php(194): Espo\ORM\QueryComposer\BaseQueryComposer->composeSelect(Object(Espo\ORM\QueryParams\Select)) #1 /home/user/domains/crm.com/public_html/crm/application/Espo/ORM/Mapper/BaseMapper.php(99): Espo\ORM\QueryComposer\BaseQueryComposer->compose(Object(Espo\ORM\QueryParams\Select)) #2 /home/user/domains/crm.com/public_html/crm/application/Espo/ORM/Repository/RDBRepository.php(103): Espo\ORM\Mapper\BaseMapper->selectOne(Object(Espo\ORM\QueryParams\Select)) #3 /home/user/domains/crm.com/public_html/crm/application/Espo/ORM/Repository/RDBRepository.php(112): Espo\ORM\Repository\RDBRepository->getById('system') #4 /home/user/domains/crm.com/public_html/crm/application/Espo/ORM/EntityManager.php(289): Espo\ORM\Repository\RDBRepository->get('system') #5 /home/user/domains/crm.com/public_html/crm/application/Espo/Core/ORM/EntityManagerProxy.php(63): Espo\ORM\EntityManager->getEntity('User', 'system') #6 /home/user/domains/crm.com/public_html/crm/application/Espo/Core/ApplicationUser.php(63): Espo\Core\ORM\EntityManagerProxy->getEntity('User', 'system') #7 /home/user/domains/crm.com/public_html/crm/application/Espo/Core/ApplicationRunners/Api.php(177): Espo\Core\ApplicationUser->setupSystemUser() #8 /home/user/domains/crm.com/public_html/crm/application/Espo/Core/ApplicationRunners/Api.php(112): Espo\Core\ApplicationRunners\Api->processRequest(Array, Object(Espo\Core\Api\RequestWrapper), Object(Espo\Core\Api\ResponseWrapper), Array) #9 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/Handlers/Strategies/RequestResponse.php(43): Espo\Core\ApplicationRunners\Api->Espo\Core\ApplicationRunners\{closure}(Object(Slim\Psr7\Request), Object(Slim\Psr7\Response), Array) #10 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/Routing/Route.php(381): Slim\Handlers\Strategies\RequestResponse->__invoke(Object(Closure), Object(Slim\Psr7\Request), Object(Slim\Psr7\Response), Array) #11 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/MiddlewareDispatcher.php(81): Slim\Routing\Route->handle(Object(Slim\Psr7\Request)) #12 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/MiddlewareDispatcher.php(81): Slim\MiddlewareDispatcher->handle(Object(Slim\Psr7\Request)) #13 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/Routing/Route.php(341): Slim\MiddlewareDispatcher->handle(Object(Slim\Psr7\Request)) #14 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/Routing/RouteRunner.php(84): Slim\Routing\Route->run(Object(Slim\Psr7\Request)) #15 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php(60): Slim\Routing\RouteRunner->handle(Object(Slim\Psr7\Request)) #16 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/MiddlewareDispatcher.php(140): Slim\Middleware\RoutingMiddleware->process(Object(Slim\Psr7\Request), Object(Slim\Routing\RouteRunner)) #17 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(107): class@anonymous->handle(Object(Slim\Psr7\Request)) #18 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/MiddlewareDispatcher.php(140): Slim\Middleware\ErrorMiddleware->process(Object(Slim\Psr7\Request), Object(class@anonymous)) #19 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/MiddlewareDispatcher.php(81): class@anonymous->handle(Object(Slim\Psr7\Request)) #20 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/App.php(215): Slim\MiddlewareDispatcher->handle(Object(Slim\Psr7\Request)) #21 /home/user/domains/crm.com/public_html/crm/vendor/slim/slim/Slim/App.php(199): Slim\App->handle(Object(Slim\Psr7\Request)) #22 /home/user/domains/crm.com/public_html/crm/application/Espo/Core/ApplicationRunners/Api.php(94): Slim\App->run() #23 /home/user/domains/crm.com/public_html/crm/application/Espo/Core/Application.php(107): Espo\Core\ApplicationRunners\Api->run(NULL) #24 /home/user/domains/crm.com/public_html/crm/api/v1/index.php(37): Espo\Core\Application->run('Espo\\Core\\Appli...') #25 {main} Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor. [] []

[esforim]

The FileManager.php exist in version v6.0.9 but for some reason it was removed from the official latest version 6.1.3.

[esforim]

After trying for about 1 hour to fix it, I fail. I decide to revert back to version 6.0.9 as it was the latest safe version for me.

Be good if anyone ran into this issue and manage to resolve it. I'm currently suspecting it due to the opcache.