EspoCRM 6.1.0 released

[murugappan]

Maximus
esforim

This is what I did:

(1) Restored my instance to version 6.0.8 - no problems
(2) Performed an upgrade and it rolled up to 6.0.10.
(3) Checked the log and whole bunch of errors shows up, similar to the previous error which was posted under https://forum.espocrm.com/forum/inst...data-truncated . (refer to screencap)
(4) Then i upgraded again which rolled-up to version 6.1.3 after a very long time.
(5) The same errors were logged in but the upgrade was successful (i think!)
(6) Delete the logs and loaded the instance. No errors logged.
(7) Did a rebuild and voila! the same error appears again.

Its odd, as i can remember that my problem surfaced when i upgraded 6.0.10 previously. Unfortunately, the upgrade from 6.0.8 steps through version 6.0.10 and seem to be creating the same issue. Thereafter any further upgrade appears to carry forward the problems. Espocrm developers need to investigate this.

I have halted at version 6.0.9. That's it for the time being because i am worried about the integrity of the database once i cross into any further upgrade which manifests the same errors in the log. I feel Espocrm need to perform the upgrade straight through from 6.0.8 or 6.0.9 to version 6.1.x instead of 6.0.10 step up.

[murugappan]

esforim, I did not face any problem like the ones you are facing. In all my instances, this is what happens:

(1) The permission error happens in all of them and is annoying and does not give me the confidence to keep the version.
(2) In the ones with larger databases, the upgrade just dies at the rebuild stage and reverts back to the previous version.
(3) Only one instance is running the version 6.1.2 with the errors stated in (1) above.

Decided to put the brakes at 6.0.8. I downloaded the full versions of 5.9.4 and 6.0.8 just in case i need to do another fresh instance. I too have 2 instances running on a Shared Hosting Service but that one has been frozen with 5.7.4. We are planning to migrate that version after we purchase additional dedicated server resources from our Hosting Provider. That core installation for that is nearly 300GB and database is 120GB. Its been running from 2018 with 250,000 cases and 500,000 emails. We need to have carefully planned migration, Risk Management and fallback action plans. Never thought i will be doing this kinda migration after 40 years, which had millions of records and on DB2 database. Nightmare.

I guess the people in ESPOCRM are busy cracking on these problems. I am assuming this as there are no responses from them.

[item]

Hello esforim,
root access needed to apache/nging/php/mysql.. and so
else, for espocrm .. we need just "www" user
contact me (private message) and i can try to help you .. or find what's is the problem

[esforim]

Hi murugappan, certainly is with that kind of filesize you facing. I don't think my issue is an current ongoing concern for the team, especially since it look like I'm the only person that having this issue. Sad to say, but I'm hoping a few other user have similar issue and might have more knowledge on how to fix it. Unlike the massive slowdown issue of MariaDB and collation table where there were 5-6 user facing same problem.

I'll get in contact with item figure out how to do remote. I have access through File Manager and MyPHPAdmin, if that is enough. Can download remote software such as TeamViewer or AnyDesk or similar software.

[Christian]

Hello,

I hope that my post on this topic is correct.

Since the update from 6.0.10 to 6.1.x there are severe performance problems with database queries.
All other options also work slower than before.
I have rebuild indexes (rebuild.php) and optimize databsase, all without improvement.
Restore to 6.0.10 is unfortunately not an option.

Example 1:
If I will export all contacts (round 9000) to csv (all fields) get the message "error 500".
Firefox abort with "error 500" after 1:40 minutes. (see attachment)
HeidiSQL processes need 207 seconds for query and in second 208 creating sort index. (see 2 attachment)
The error log (DEBUG) is empty for the time of the query.

Example 2:
If I will mass update all accounts (round 2600) get the message "timeout".
Firefox abort without info after round 60 seconds. (see attachment)
HeidiSQL processes need 53 seconds for query. (see attachment)
The error log (DEBUG) is empty for the time of the query.

Can you help me, please!


Windows 2012 R2
Apache
MariaDB 10.4.17
(InnoDB, all table "uft8mb4_unicode_ci")
PHP 7.4.15
EspoCRM 6.1.4
(updates from 6.0.10 to 6.1.0 to 6.1.1 to 6.1.2 to 6.1.3 to 6.1.4 all over cli)

[Christian]

Hello murugappan,

my PHP config with all set values (PHP NTS run as fastcgi on Apache):

[PHP]
engine = On
short_open_tag = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
serialize_precision = -1
open_basedir = [Path to htdocs and cache folder]
zend.enable_gc = On
zend.exception_ignore_args = On

expose_php = Off
max_execution_time = 180
max_input_time = 180
max_input_vars = 5000
memory_limit = 256M

error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
error_log = [Path to log file]

variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 25M
default_mimetype = "text/html"
default_charset = "UTF-8"

extension_dir = [Path to php\ext folder]
sys_temp_dir = [Path to apache cache folder]
enable_dl = Off
cgi.force_redirect = 1
fastcgi.logging = 1

file_uploads = On
upload_tmp_dir = [Path to apache cache folder]
upload_max_filesize = 25M
max_file_uploads = 20

allow_url_fopen = Off
allow_url_include = Off
default_socket_timeout = 60

extension=curl
extension=gd2
extension=imap
extension=mbstring
extension=exif
extension=mysqli
extension=openssl
extension=pdo_mysql

[CLI Server]
cli_server.color = On
[Date]
date.timezone = "Europe/Berlin"
[mail function]
mail.add_x_header = Off
[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
[OCI8]
[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
[bcmath]
bcmath.scale = 0
[browscap]
[Session]
session.save_handler = files
session.save_path = [Path to php cache folder]
session.use_strict_mode = 1
session.use_cookies = 1
session.cookie_secure = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_httponly = 1
session.cookie_samesite = Strict
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check = hhl.de
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.sid_length = 48
session.trans_sid_tags = "a=href,area=href,frame=src,form="
session.trans_sid_hosts=""
session.sid_bits_per_character = 6
[Assertion]
zend.assertions = -1
[Tidy]
tidy.clean_output = Off
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir= [Path to php cache folder]
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
[ldap]
ldap.max_links = -1

---

to Example 1:
Yes, I will export all contacts with fiels to csv.
Before the Update to 6.1.x it works fine.
If I export all contacts with only 3 fields get the same error.

Example 3:
If I search a last name in contacts and get "timeout" and search afterwards again get the right results.
I need 2 searches to get a result.

Thank you for your support.

[murugappan]

Hi @Christian,

Try these:
max_execution_time = 1200
max_input_time = 1200
max_input_vars = 80000
memory_limit = 512M

These do affect the amount of resources available for php to execute the job.

[Christian]

Hi murugappan,

I ran yesterday the script to update "uft8mb4_unicode_ci" (https://forum.espocrm.com/forum/inst...rade#post61212) to test it.

According to the database, the tables were already in uft8mb4_unicode_ci, but the script had the desired effect!

After the script are any performance problems are resolved.
All examples mentioned (1-3) work again without errors.

Thanks for your support.

[emillod]

Hello guys,
i can confirm that changing collation for ID columns to utf8mb4(we've changed to utf8mb4_unicode_ci) fixed all issues with performance
Thanks yuri

[marcusquinn]

Really nice work on this! Thought I'd share my design so far with menu Grouping, in the spirit of sharing & caring It's based a lot on being generic to all industry organisations but with some specifics for a couple of niches I'm working on.

I've ordered the main workflow from Campaigns to tasks with the rainbow colours red to violet from bottom to top as business goes from outreach to actual, and Calendar & Email on top as the hub of all communications. I like logical structures and thinking about user interface intuition for most commonly used elements and ordering by pathway through business and delivery

I have a lot of sub-menu items under multiple top levels too, and will use Teams & Roles to filter who sees what, so that it is focused for each Role.

Role-based permissions management in any system makes me very happy - so my compliments again to whoever design these aspects of the framework, it makes multidimensional data design so much easier when someone's been so thoughtful with the framework!

[esforim]

It always good to see example to be inspire by (I always have trouble with group naming). But what is beneath those menu? Be good if you can show those.

[emillod]

Wow! I didn't know anyone had more modules than us in EspoCRM. I will take this personally and take the challenge

[marcusquinn]

Admiring all your work here and certainly interested in collaborations. I'm going full ERP with this, and have extensive experience with Microsoft Dynamics NAV, Odoo, Magento, Wordpress, Woocommerce, Freshdesk, Freshworks, Appogee HR and many, many other apps over 20 years in this business, so my instance will have inspiration from all of those, and be better Will upload more screenshots but this forums has a fair few limits, so bear with me...

[marcusquinn]

[marcusquinn]

[marcusquinn]

[marcusquinn]

[marcusquinn]

I'm only about 2 months into Espo, so it's work in progress. Lots done and lots still to do. I'll get a test clone instance setup for collaboration if anyone is interested.

Most of my work is business data structure and flow design, and everything is with respect to best-practice for regulations, compliance, accounting and auditable recordkeeping.

The end result will be an instance that anyone can run and document their business setup, structure and team with. All the things common across all organisations will be generic, and some industry-specific things will only show if Users have the appropriate team assigned.

Also, love the Portals feature for external collaboration with various partners & 3rd parties, so designing a lot for that too.

[esforim]

The crazy amount of submenu, was only expecting a few per category to be honest. I see some of it duplicate in submenu (e.g. Multiple Property, multiple call). And reading the main category, it look like it meant to be for different services/business.

It this just your "all-in-one" portal and depending on the business they only really use the relevant one applicable to them? I can see it be quite overwhelming if you activate it all on a newly installation.

Personally my skill in coding is limited, but I love to give it a try for feedback.

Perhaps it might be best to create a new thread if further discussion into this, else it just get unknown in thread v6.1.0

[marcusquinn]

The duplicates are because some objects apply to many departments. With the use of Teams & Roles, department users & clients would only need to see what's relevant to them.

Sure, I'll start a separate thread at some point. I've used almost every ERP/CRM system over the years, so this is being built out to follow all the best-practice and conventional standards across all of them - naturally with the advantage of open-source meaning unrestricted when many of the enterprise systems charge for things like tables as *extras*.

[item]

marcusquinn,
Do you understand why i give name "einstein" to Yuri ?

On front-end, we need absolutly "tab".. organise panel as tab.
On back-end (entityManager), we need absolutly a solution for middle table (one2many or many2many) can act as out-of-box entity. (possibility to add field, layout, ...and so)

one2many => something like phone field, reminders fields..
many2many => add custom field but can place in layout

Regards

[marcusquinn]

It's an appropriate nickname for sure!!

Looking forward to doing so much more with this framework, and happy to meet so many of you already experienced and enthusiastic for the possibilities this all creates

[murugappan]

Great New folks! My problem has resolved and the upgrades are fast and smooth without any errors. The problem was the permissions parameter in the config.php. All my instances, not sure why or how, show the following:

user => 1003,
groups = 1005

After a fantastic help from Maximus who debugged the problem, i installed a fresh version 5.9.4 and found that the permissions were:

user => 1000,
groups => 1002

Based on this, i changed the permissions and now all the upgraded completed fast and smooth without any errors.

MY HEARTIEST THANKS TO Maximus FOR HIS PATIENCE WITH ME AND ALL THE HELP.

[murugappan]

A word of reservation. After more testing, i found that this may only work if the instance has not been upgraded to version 6.0.10. Once you sink into versions > 6.0.10, the problem continues to exists. I have raised this with Maximus again. I do have 3 instances (test, staging, and production) relating to one app with this problem.

In one instance, after smooth upgrading to 6.1.4, there was a problem/error with the Scheduled Jobs on the Group Email Accounts. I found that a Group Email Account and the scheduler has some problem. All i did was to remove the "faulty" Group Email Account and created a new one in its place. Everything went on fine and cron is fetching the email correctly now. Phew!

[marcusquinn]

Erm, spam? ^

[murugappan]

Hi all,

With a lot of tireless help from Maximus, we managed to resolve the problem. The problem was related to a mismatched owner and group id in the Espocrm instance config .php file and linux file ownnerships. This is what we did to resolve the problem:

(1) cd public_html/ ===> Web root
(2) stat tigascrm =====> tigascrm is the instance root folder
(3) I get "Uid: ( 1001/tigasdispensing) Gid: ( 1003/tigasdispensing)" as output
(4) edit the tigascrm->data->config.php
(5) locate the code that says 'defaultPermissions'
(6) Change the value next to "user' to 1001
(7) Change the value next to 'group' to 1003

Yipeee! All issues and errors gone. Log is clean now. A lot of thanks to the great help from Maximus.

[esforim]

I'm curious, does permissions of these show all clear in the "System Requirements"? Should there be a feature request that permission can be check for these important folder/file?

[murugappan]

Hi esforim , there is short documentation on this requirement but most of us ignored or did not know if we had to do anything about it. The link is https://docs.espocrm.com/administrat...ion/#ownership. I saw that many times but was did not know how to do it or what the consequences would be .Your right, when these sorts of requirements become a show-stopper, it perhaps the right time for the documentation to be updated and it will be really good if there is some in-built check on system changes in the upgrade itself.