Email Fetching Duplicate Messages

[joy11]

I just thought i'd give an update on this issue. This looks like it's been completely resolved. I'm running v5.0.5 now and haven't seen duplicates since the update. Thank you for all your hard work yuri !

[yuri]

Hi,

Have you decreased max portion size? The aim is a portion fetched in less than 2 hours.

Also you can bump up jobPeriod.

[cardmaverick]

I just dug in deeper here, looking at message headers in my workspace email client to see if messages with null message-id fields were missing the 'Message-Id' header, they are.

I also upgraded the importer.php file and reverted back to old database index / column settings for message-id column - tons of duplicates entering the system.

[cardmaverick]

So I just got off the phone with Godaddy - I'm not sure if this is the issue or not, but my email is very old, it used to be on a POP3 Server but they migrated everyone to an IMAP server. So a lot of my mailbox at one point in time was handled by POP3, I don't know if this affects email formatting but I figure it's worth mentioning. I'm looking at having to just cut ties with the box and move to something more modern and *clean slate*.

[cardmaverick]

I was looking at your importer code - are you looking for duplicates based strictly on message ID?

protected function findDuplicate(Entity $email)
{
if ($email->get('messageId')) {
$duplicate = $this->getEntityManager()->getRepository('Email')->where(array(
'messageId' => $email->get('messageId')
))->findOne();
if ($duplicate) {
return $duplicate;
}
}
}

A system of hashing out the actual message content into a 255 character key might work better, I saw a ton of NULL message ID's before disallowing null IDs in the database message id column - unless there is an issue deeper in your program that is erasing them.

[joy11]

The emails do have identical message_id fields in the database. I have the max email portion size set to 20 on personal and 20 on group. I turned off most of my workflow jobs to try and improve performance. This morning I made a change to the 'minExecutionTime' => 60, (before I saw your post below) It didn't seem to hurt anything during the day and I had only 1 duplicate for several hours. This evening I did upgrade to 5.0.3 and have seen several duplicates since the upgrade. I will wait for next release now since my upgrade is clean.

[yuri]

I'd recommend you to wait for the next release. We are going to improve scheduled jobs implementation.

[yuri]

minExecutionTime shouldn't be touched. You can bump up jobPeriod. Also set attempts = 1

[joy11]

Great thank you. My files have this code in this 'cron' section also:

/** Min time (in seconds) between two cron runs. */
'minExecutionTime' => 50,

Do you know if this is the time between two of the same job attempts? If it is, I'll bump this execution time up and that should help.

[yuri]

Ok. Now we are working on this issue. There are two points:

1. Make an ability for jobs to be executed in a long period. More than 2 hours like it is now. We will need to control PID. At lease we can pull it off for unix-based systems.
2. Prevent email duplicates that can occur in extreme conditions, e.g. when multiple processes are executed simultaneously and server run out of resources.

[tanya]

The default is in application/Espo/Core/defaults/systemConfig.php, but you can override the default value in data/config.php

[yuri]

Do these duplicates have an identical message id?

What max email portion size is set in Administation > Inbound Emails?

Thanks.

[joy11]

What file should this go in? I didn't see it in data/config.php. Is there someplace else?

[joy11]

I added this update and it doesn't seem to help the issue. I have 2 group emails and approximately 30 personal emails in the system. The personal emails have inbox and sent items monitored. We're using Gmail for business.

I ran it for approximately 3 hours today and noticed in my mySQL processes (live view) there were several (7 or so) open sleeping connections with +100 time listed. I was still getting duplicate and triplicate emails in the system and sending from the system. It appeared to be mainly on personal emails. Gmail was also having some issues this afternoon, but when I reverted back to the original Importer.php it seemed to improve.

I've been having random duplicate emails for a while, but very few and far between. Today, before the change to Importer.php I had 15 duplicates within 1 hour. I changed the Importer.php and was fine for about an hour then had 17 duplicate/triplicates the following hour.

I was unable to find the spot in the data/config.php to change anything about the cron execution. Maybe I'm looking in the wrong spot?

[yuri]

Applying inquie key we can't afford for now for the next reasons:
1. Users can already have null and dupllicates in their database.
2. Upgrade will take way too long, since email table are usually big.

You can apply it manually for your database. Exception will be catched and no fatal error should occur.