upgrade stuck on rebuilding

[Emmker]

V8.4.2 to 9.0.6

I run php command.php upgrade and starts but stuck on rebuilding.

I tried to hard rebuild first, but that failed too.

I can rollback the server, so I can try many suggestions.

system requirements on picture.

Server:
8 CPUs (i7 10700)
16GB ram
more than 500gb free disc space.

[yuri]

How long did it take before you terminated the script? If you have a lot or records in some tables, it can take a while as it may add some indexes.

[Emmker]

I waited around 5 minutes for the upgrade.
And 7 minutes for the hard rebuild before I terminated the script.
I have a LOT of emails. around 400.000

[yuri]

It can take much more. E.g. 20 - 30 mins. Depends on your hardware and configuration.

[Emmker]

I will re do it now.

[Emmker]

I have it on rebuilding for 1h

[yuri]

Did you disable cron beforehand? Enabled cron can slow down rebuild.

[Emmker]

Cron disabled.
I even tried, with all the 3rd party extensions uninstalled.
Same result.
Yesterday, I left it running for 3 hours. Nothing.

Is there any way I can see the progress of the script to troubleshoot?

[lazovic]

Hi Emmker,

Please tell me, are there any errors in the logs of your EspoCRM instance? They also may be occurring on the server side.

[Emmker]

Nothing relevant.
Nothing server side too.
The only thing that I have observed, is that during the upgrade I have 30-40% cpu till the last "rebuilding" and then 1-2%
So I think it just stops.

[yuri]

Likely it's an SQL query executing long. It could be a table column creation. You can enable debug mode and enable SQL logging to figure out what query executes long.

[Emmker]

I will try to do that today and report back

[Emmker]

I run with debug and when the stuck rebuilding... show I get

[PHP][2025-03-24 12:11:20] INFO: Schema, query: CREATE TABLE app_secret (id VARCHAR(24) CHARACTER SET utf8mb4 NOT NULL COLLATE `utf8mb4_unicode_ci`, name VARCHAR(100) CHARACTER SET utf8mb4 DEFAULT NULL COLLATE `utf8mb4_unicode_ci`, deleted TINYINT(1) DEFAULT 0, value MEDIUMTEXT CHARACTER SET utf8mb4 DEFAULT NULL COLLATE `utf8mb4_unicode_ci`, description MEDIUMTEXT CHARACTER SET utf8mb4 DEFAULT NULL COLLATE `utf8mb4_unicode_ci`, created_at DATETIME DEFAULT NULL, modified_at DATETIME DEFAULT NULL, delete_id VARCHAR(17) CHARACTER SET utf8mb4 DEFAULT '0' NOT NULL COLLATE `utf8mb4_unicode_ci`, created_by_id VARCHAR(24) CHARACTER SET utf8mb4 DEFAULT NULL COLLATE `utf8mb4_unicode_ci`, modified_by_id VARCHAR(24) CHARACTER SET utf8mb4 DEFAULT NULL COLLATE `utf8mb4_unicode_ci`, UNIQUE INDEX UNIQ_NAME_DELETE_ID (name, delete_id), INDEX IDX_NAME (name), INDEX IDX_CREATED_BY_ID (created_by_id), INDEX IDX_MODIFIED_BY_ID (modified_by_id), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE `utf8_unicode_ci` ENGINE = InnoDB
[2025-03-24 12:11:20] INFO: Schema, query: CREATE TABLE user_reaction (id VARCHAR(24) CHARACTER SET utf8mb4 NOT NULL COLLATE `utf8mb4_unicode_ci`, type VARCHAR(10) CHARACTER SET utf8mb4 DEFAULT NULL COLLATE `utf8mb4_unicode_ci`, created_at DATETIME DEFAULT NULL, user_id VARCHAR(24) CHARACTER SET utf8mb4 DEFAULT NULL COLLATE `utf8mb4_unicode_ci`, parent_id VARCHAR(24) CHARACTER SET utf8mb4 DEFAULT NULL COLLATE `utf8mb4_unicode_ci`, parent_type VARCHAR(100) CHARACTER SET utf8mb4 DEFAULT NULL COLLATE `utf8mb4_unicode_ci`, UNIQUE INDEX UNIQ_PARENT_USER_TYPE (parent_id, parent_type, user_id, type), INDEX IDX_USER_ID (user_id), INDEX IDX_PARENT (parent_id, parent_type), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8 COLLATE `utf8_unicode_ci` ENGINE = InnoDB
[2025-03-24 12:11:20] INFO: Schema, query: ALTER TABLE email ADD send_at DATETIME DEFAULT NULL, ADD group_status_folder VARCHAR(7) CHARACTER SET utf8mb4 DEFAULT NULL COLLATE `utf8mb4_unicode_ci`[/PHP]

I do not know if it is related or not, but for months now I cannot add a new field to the email entity.
When I try to do that, it gives me a timeout.

I also included the debug log too

[yuri]

For some reason your MariaDB cannot add a new column or it takes way too long. I have no idea why. Was your CRM in use while upgrading?

[Emmker]

No. In fact, this is the development instance, on which we test the changes and updates first before production.

any ideas to repair it? Or at least a direction to start?