ERROR: Json::encode ()

[override]

Hi,

I have an issue with displaying emails which have some special characters like ® or ™

Bad server response in client and in logs

ERROR: Json::encode():Malformed UTF-8 characters, possibly incorrectly encoded - stdClass Object ( [id] => 603cecd205367d5fc........

I have two machines, both are using latest version of esporm, also one uses MySql while another uses MariaDB.
Also, I noticed that body_plain for email is NULL for that specific email. I updated body_plain with body content and it can be parsed/shown in browser.

Any suggestion on how to solve this issue?

Best regards

[override]

I can send an email which causes aforementioned error, if someone is willing to troubleshoot it...

[yuri]

Could be an issue with a charset in database.

If you provide an email original I will take a look.

[override]

Attached is dumped body of an email that can't be parsed.
Database character set is utf8, collation database utf8_general_ci.
body field of an email table has collation utf8mb4_unicode_ci

On older espo and mysql versions email couldn't be imported to database, it reported some kind of error which is related to that ® symbol.
Now, database doesn't have errors and body contains email content but it can't be read via browser.

What is also interesting is that symbol is shown differently if sent from gmail
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| body |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| <div dir="ltr"><span style="color:rgb(37,44,47);font-family:Helvetica,Arial,Verdana,sans-serif;font-size:13px;background-color:rgb(235,244,249)">® </span><br></div>
|
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I also tried
update email set body = replace(body, '&reg;', '®') where id='605315b14e17a375d';
and it works,.... so &reg; is cause of problem....

After that it appears that only semicolon is actual problem
so if ; in &reg; is omitted so that only &reg is between <sup> tags it works and renders symbol but with ; at the end it doesn't.

I suppose, one kind of a solution to this problem would be custom hook that will check each email and updating content....

I appreciate if you have any, cleaner, solution.