Remove Duplicate (process via cron job when number is too high) - Importing

[rabii]

When importing big files with the ability find existing duplicate, an error timeout will be produced when trying to remove duplicate (couple thousand 5k in my use case). I think it will be nice to have the remove duplicate processed via cron job when the number hits a certain level (maybe configurable option).

Thanks

[dimyy]

Most likely, duplicates are searched for by non-indexed fields. I recommend looking at the log of long time queries in the database.

https://mariadb.com/kb/en/slow-query-log-overview/