method Exit Takes Ages

[pigg]

Hello All!

 

I have a method in a class I am working on that imports an array of data into a database. The method takes quite a large array (26,000+ items, each with about 15 separate pieces of information to update). This method takes a while, but that is to be expected. What takes ages is returning from the method. Let me explain...

 

The method has logging messages through out that I monitor via an SSH terminal (tail -f msg.log) so that I can see the progress of the method. The message just before the method is closed is logged to the error file I'm monitoring but the error message in the line after the method call never gets displayed!

 

Here is some simple code that might explain:

 

<?php

class MyDbImporter {
public function importArray($data) {
   _log('Importing starting');
   $this->_import();
  _log('Importing finished');
}
}


$data = getDataFromSource();

$db = new MyDbImporter();

$db->importArray($data);

_log('Import successful. All Finished!');

 

Although the code above is a simplified version of what I have, it hopefully illustrates my problem. Basically, the 'Importing finished' message from within importArray is displayed but the 'Import successful. All Finished!' message never displays.

 

My only idea is that this could be PHP cleaning up memory or something before exiting the method.

 

Does any one have any ideas or links that could help me?

 

Thanks in advance!

[Psycho]

Well, if the 'Importing starting' message is displayed and the 'Importing finished' message is not displayed then something is happening within the call to $this->_import(); that is halting the script. You didn't show that code, so there's no way to tell what the problem is. But, I do have one question/suggestion:

 

You say the process is taking a long time. Yes, 26K records is a lot. But, that also means that introducing even a small piece of inefficient code will have significant consequences. If you are using this data to insert records into a database you should definitely be concatenating all the INSERT records into ONE insert query. Running 26K queries is incredibly inefficient and changing ti to do one query will likely cut the processing time to a fraction of what it is now.

[pigg]

Hi Physco,

 

Thanks for the quick reply.

 

The 'Import Starting' and 'Import Finished' messages are both displayed, it is the message at the very bottom of my code (after the importArray method has returned) that isn't displayed.

 

Although the code doesn't match my code (my code is posted over several different files) it highlights what's happening. Basically, a method that takes an array imports it successfully gets to the end but never seems to actually return from the method.

 

I see your point about importing the data and concatenating my queries however the system I'm working with uses EAV so I'm unable to to do so.

 

 

[Psycho]

The 'Import Starting' and 'Import Finished' messages are both displayed, it is the message at the very bottom of my code (after the importArray method has returned) that isn't displayed.

 

Oh, ok. The text-area display in my browser was just a tad too small to show that last line and I didn't notice to scroll down.

 

So you have these two lines

$db->importArray($data);

_log('Import successful. All Finished!');

 

The importArray() method has logging events that are working, but the log message right after that method is not getting registered. I would again go back to the premise that something in the importArray() is causing the script to terminate completely. I assume that the actual importArray() method has a looping structure in it where the successful logging is taking place. Have you tried implementing a log message in that method right after the loop before control is returned to where the method was called? I don't know how many lines of code you may have after the loop completes, but you could add log messages between each line to see where the last one succeeds. That's really the only advice I can suggest - add as much debugging as you can to pinpoint the problem.

[pigg]

Yeah, I have a log message at the start and end of each loop iteration. What's wierd is that the log message at the end of importArray is the very last line of the method and executes successully as I can see the output in my log file. The next line after this is the closing bracket of the method, meaning technically there is nothing to stop the method completing.

 

What leads me towards it being an issue with PHP having clear the memory/garbage is that if I reduce the amount of data items that need to be imported, the method returns successfully and all messages are displayed.

[xyph]

Could be a memory issue, or execution time?

 

Are you logging/tracking PHP errors?

[PFMaBiSmAd]

Without all the code that reproduces the problem, including your _log() function, it's not directly or quickly possible to help you find what the problem is.

 

If you don't want to post all your code that reproduces the problem, make up a sub-set of the code that still produces the problem when you run it, then post all of that sub-set of code. I'm guessing your _log() function has some variables brought into it using the global keyword that are part of that class and they don't exist when you call that function outside of the class.