Large IMAP loop starts to hang around 20,000 emails...

[youneek]

Does anyone know why this starts to go so slow or a way to get this script working better? I have tested retrieving the higher number messages and can retrieve them fine individually...

 

<?
set_time_limit(0);
$imap = imap_open("{imap.server.loc}", "emailaddy", "pass")or die(imap_last_error()."<br>Connection Failure!");
$count = imap_num_msg($imap);

$fh = fopen('results.csv', 'w') or die("can't open file");

for($i=1; $i<=$count; $i++) {
$header = imap_header($imap, $i); 
$prettydate = date("m/d/Y", $header->udate);

$user = strtolower($header->sender[0]->mailbox);
$host = strtolower($header->sender[0]->host);

if(!empty($user)){
	fwrite($fh, $user.'@'.$host.','.$prettydate."\n");
}
}

imap_close($imap);
?>

[youneek]

I have tested the tool. If I stop and reset it to start at 20,000 it runs at full speed again... Any suggestions?

 

for($i=20000; $i<=$count; $i++) {

[JeanieTallis]

Maybe, its the power of the code. The weaker the code, and yet the more emails it sends out, the slower it'll take as its dealing with more loops. I'm not sure. This isn't what I study at.

[redarrow]

you can not use php to send as many emails as you want, you have to use a email sending program.

 

php was not design to send multiple emails like a huge mailing list.

 

You can use the sleep() function, but like i say php is not used for a mailing list.

 

your php sever will eventually blow up.

[.josh]

your php sever will eventually blow up.

 

holy crap! I bet that sure would piss the guys off at the data center.  Would it make other servers connected to it blow up too??

[monokrome]

PHP doesn't have a garbage collector...

 

With that said, my guess is that you're using way too much memory maybe. Unset doesn't free memory so that wont help, and you're using (40,000*SizeOfUserVariables)kb by the end of the script. I would probably not use PHP to send 40,000 emails in a single refresh, regardless.

 

[MadTechie]

PHP doesn't have a garbage collector...

 

Erm.. Yes it does!

PHP does garbage collection at 3 points

1. When you use unset() or other cleanup options!

2. When you leave a function (php clears any variable that leaves scope)

3. When the script ends (php clears any variable that leaves scope)

 

[corbin]

PHP doesn't have a garbage collector...

 

With that said, my guess is that you're using way too much memory maybe. Unset doesn't free memory so that wont help, and you're using (40,000*SizeOfUserVariables)kb by the end of the script. I would probably not use PHP to send 40,000 emails in a single refresh, regardless.

 

 

 

Besides MadTechie's points, the OP is also resetting variables.  Your point would only be valid if he were making a new memory space for each thing.  (I.E. if he were cramming it all into an array or something.)

 

 

 

 

 

 

@OP:

 

 

I would say that the problem is either the imap_header() call or the 20,000 fwrite calls ;p.

 

 

Checking if it's the imap_header() call could be kind of difficult, so the easiest way would be to check if it's the fwrite call ;p.

 

 

You could try something like:

 

 

$content = '';
$f = fopen(...);
for($i = 0; $i < 20000; ++$i) {
    $content .= ...;
    if(strlen($content) >= 8192) {
        //content is 8KB
        fwrite($f, $content);
        $content = '';
    }
}

 

[youneek]

It's not sending emails and is only retrieving the headers to write to a file. I run it locally and can see it's using hardly any ram... I have another script that can write 40,000 lines using fwrite without any issues. Is there a more effective way to do what I am doing?

[corbin]

Well, if it's not the file stuff, then it's most likely $header = imap_header($imap, $i).

 

 

 

I wonder if there is an alternative IMAP library....

 

 

Just of curiosity, try running a simple benchmark in which you call $header = imap_header($imap, $i); and do nothing with it.  Try doing it with 20,000+ iterations, and 5,000 iterations and see if there is a difference in requests/second.