marz Posted September 21, 2010 Share Posted September 21, 2010 I have a very weird problem on my PHP5.3/Apache/MySQL based system on WinXp Pro. I have written php script which basically scans every directory in my local hard disc, checks which files are images, sets a counter of the number of images found per folder and saves this count in a MySQL table. To put you in the picture, we are talking about 1000 folders with about 8-10pics each on average 9000 files. On the previous PHP 5.1 / IIS system, (same PC) this used to work moderately well (took a nice 6 mins) and I had to include the command set_time_limit of about 3seconds in the looping to avoid execution timeout failures. (also worked by setting a high value of 10mins before the loopping) ANyway, now I run the exact same script on the new Php5.3 and have the problem that the code starts operating quite fast and gradually it slows and slows until eventually it is dead slow and have to quit internet explorer. For exampe, the first 3000 files are scanned in just 60seconds, then the next 1000 in the next 4 mins, and eventually drops to about 100files or less files per minute. I also noticed that the CPU consumption of iexplorer.exe is 90-99%, and the window of Internet explorer is very slugish with a temp '(not responding)' messages on the top window bar. WHat can I do and test? Is it the code itself, the PHP5.3 or something in php.ini or apache conf. file? Buffers? memory? WHat? gonna crazy! basically, the code scans a dir (location of dir taken from Mysql) puts file contents in an array, analyse the array for .jpg/.gif (and other parameters specific to pics, counts the valid pic files in a variable, puts this in a MySql databse, loops for next folder. The code is a kind of tool to have a count of my pics in my folders and never used remote server/online but only local. Thanks for your time Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/ Share on other sites More sharing options...
the182guy Posted September 21, 2010 Share Posted September 21, 2010 I would call the sleep(1) function inside the looping after every couple of hundred files or so. That will halt execution for a second (or however many) and it will release the CPU which should stop the 'not responding' stuff. Play around with it. Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1113751 Share on other sites More sharing options...
PFMaBiSmAd Posted September 21, 2010 Share Posted September 21, 2010 A picture is worth a 1000 words. Posting your code would be the quickest way of getting an answer because it would show how you are querying the database, how you are outputting the status information, how you are managing data/variables, if you are doing something that is php/IIS/Apache specific... Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1113755 Share on other sites More sharing options...
marz Posted September 22, 2010 Author Share Posted September 22, 2010 Here is the routine - I am no PHP expert, so the coding is quite basic: :shrug] $totalimages = 0; // to store the number of images found in folder mysql_conn(); // Custom function to connect with MySQL $db_query = "SELECT FAML, ID, species FROM sppindex order by species"; $db_result = mysql_query($db_query) or die ('Error in query: ' . $db_query . mysql_error()); while ($db_data = mysql_fetch_array ($db_result)) { $ID = trim($db_data['ID']); // get ID code from db $sp = trim($db_data['species']); // get species name from db $faml = trim($db_data['FAML']); // get family name from db $picdir = "../".$faml.'/Pics/'.$ID.'/'; // build path in local hd $spname = str_replace(" ", "_", $sp); // replace spaces with _ in species name $spname = str_replace(".", "", $spname); unset ($picfiles); // array for storing filenames from the working directory print '<u> Working image dir: <font color="blue">'. $picdir . '</font> Species: <font color="blue">' . $sp . '</font> ID: <font color="blue">' .$ID . '</font></u><br><br> <font size="2">'; $dirhandler = opendir($picdir); while (false !== ($filename = readdir($dirhandler))) { $picfiles[] = $filename; } sort($picfiles); // all files in Pics/ directory are in array $picfiles // print_r ($picfiles) ; print "<br>"; $piccount=0; // counter for number of valid jpg/gif files found in directory foreach ($picfiles as $filename) // examine every array value, that is every filename in Pics/ { $filename = basename($filename) ; if (is_dir($picdir . $filename)==true) // skip if a dir is found {print "<font color=blue><b>" . $filename . "</font></b> <font color = red> - is a Dir (skipped...) </font><br>";} else // if filename, procede { $fsize = filesize($picdir . $filename); // get filesize $img_array = getimagesize ($picdir . $filename); // get image info // print_r($img_array); print "<br><br>"; $filetype = (substr($img_array['mime'], 0, 5)) ; $imgtype = (substr($img_array['mime'], 6)) ; print "Analysing file : <font color=blue><b>" . $filename . "</font></b>"; // start series of tests for finding valid images if (filetype($picdir . $filename) !== "file") // REF A {print "<font color = red> - this is not a file (skipped...) </font><br>";} elseif ( (strpos(strtolower($filename), ".jpg")==false) AND (strpos(strtolower($filename), ".jpeg")==false) AND (strpos(strtolower($filename), ".gif")==false) ) {print "<font color = red> - extension not jpg/gif/bmp/png (skipped...) </font><br>";} elseif ( $filetype != "image") {print "<font color = red> - File not really an image (skipped...) </font><br>";} elseif ( $fsize < 8000) {print "<font color = red> - Filesize only [" . $fsize ."bytes] Does not qualify for a valid image (skipped...) </font><br>";} elseif ( strpos(strtolower($filename), "_t.") !=0 OR strpos(strtolower($filename), "_t2.") !=0 OR strpos(strtolower($filename), "_t3.") !=0 OR strpos(strtolower($filename), "_t4.") !=0 ) {print "<font color = red> - This is a thumbnail image (skipped...) </font><br>";} else { $piccount++ ; print "<font color=green> - " .$imgtype. " Image found (" . $img_array[0]." x ".$img_array[1].") Image Count: ".$piccount."</font><br>";} } // End of if/else for is_Dir } // End of Foreach loop print '<br><font size="+1"> Total number of images for profile <font color="blue">' . $sp . "</font> is <b> " . $piccount. "<b></font> " ; closedir(); $img_array= array(); $sql_edit = "Update sppindex Set rem3 = '$piccount' Where ID = '$ID' "; if (mysql_query ($sql_edit)) {print "<br> Inserted in DataBase sppindex !";} else {die ('Error in query: ' . $sql_edit . mysql_error());} $totalimages = $totalimages + $piccount; print "<br> $totalimages "; print "</font><br><hr><br><br>"; set_time_limit (100); // this will extend execution time 10 second every loop, so as it will never stop due max exec time reached. (tool takes 4 mins) } // End of While Loop print "<br><hr><br><h2><center><font color=red>Total number of photos in Maltawildplants.com is: " . $totalimages . " photos. </font></h2> "; mysql_free_result($db_query); mysql_disc(); Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1113969 Share on other sites More sharing options...
marz Posted September 22, 2010 Author Share Posted September 22, 2010 182guy - thanks mate - I have not had time to try the sleep() function - I just posted this before going to work. It might work as from the symptoms observed in iexplorer, it is like there is a gradual build of hangup. :-\ PS - the textbox to write posts in this forum is very annoying. When you reach the bottom it moves up away from the cursor making editing almost blind. Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1113970 Share on other sites More sharing options...
marz Posted September 22, 2010 Author Share Posted September 22, 2010 "A picture is worth a 1000 words." That what I suppose too ! Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1114233 Share on other sites More sharing options...
Mchl Posted September 22, 2010 Share Posted September 22, 2010 Are you running this from your browser? Don't. Run it from command line. Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1114269 Share on other sites More sharing options...
samshel Posted September 22, 2010 Share Posted September 22, 2010 http://theserverpages.com/php/manual/en/function.readdir.php An Extract. Might be useful ! Quote IT IS VITALLY IMPORTANT that when using opendir and then scanning with readdir, that you test opendir to see if it has returned a valid pointer. The following code does not : <?php $dir = opendir($mydir); while((false!==($file=readdir($dir)))) if($file!="." and $file !="..") unlink($mydir.'/'.$file); closedir($dir); ?> if $mydir is not a valid directory resource (for example, the directory in question has been deleted), then the code will loop infinitely. Of course, it'll stop eventually if you have set a max execution time, but in the meantime it will visibly slow the server. Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1114287 Share on other sites More sharing options...
Psycho Posted September 22, 2010 Share Posted September 22, 2010 I'm sure there are other efficiencies to be had, but the one that jumps out at me is that you are running queries in a loop. Doing that is horribly inefficient. There are two possible solutions I see: 1. Use an INSERT statement with ON DUPLICATE KEY UPDATE instead of just an UPDATE statement. You would then gather the data for the query within the loop and after the loop completes run a single query. Example: In the loop change this line $sql_edit = "Update sppindex Set rem3 = '$piccount' Where ID = '$ID' "; To this $values[] = "('$ID', '$piccount')"; Then after the loop has completed, create a single query like this $sql_edit = "INSERT INTO sppindex (ID, rem3) VALUES " . implode(', ', $values) . ON DUPLICATE KEY UPDATE rem3=VALUES(rem3)"; 2. The other option is similar. But, instead of using ON DUPLICATE KEY UPDATE you create a single UPDATE statement that will do the same thing. Example: In the loop change this line $sql_edit = "Update sppindex Set rem3 = '$piccount' Where ID = '$ID' "; To this $cases[] = "WHEN '$ID' THEN '$piccount'"; Then after the loop has completed, create a single query like this $sql_edit = "UPDATE `sppindex` SET `rem3` = CASE `ID` " . implode("\n", $cases) . "END"; I'm sure one method is more efficient than the other. You could do some testing to find out which is better. Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1114301 Share on other sites More sharing options...
BlueSkyIS Posted September 22, 2010 Share Posted September 22, 2010 You might also consider reducing the number of writes to the browser. That might be part of the reason IE is hogging resources: just holding all that output in memory. ... and if IE is eating all your resources, Apache/PHP/MySQL will have to slow down to accommodate. Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1114308 Share on other sites More sharing options...
marz Posted September 26, 2010 Author Share Posted September 26, 2010 I will test everything you said in a while - the sleep (1) function however didn't work. What's strange is that it used to work quite fine on the prev installation and also on the laptop. If something comes in mind related to php.ini , let me know Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1115821 Share on other sites More sharing options...
kickstart Posted September 26, 2010 Share Posted September 26, 2010 Hi I would be inclined to use INSERT / ON DUPLICATE KEY UPDATE as suggested above, but batch up the inserts rather than doing them one at a time. All the best Keith Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1115827 Share on other sites More sharing options...
marz Posted September 26, 2010 Author Share Posted September 26, 2010 Update 1: I have commented out the part where there is the mysql update: // $sql_edit = "Update sppindex Set rem3 = '$piccount' Where ID = '$ID' "; // if (mysql_query ($sql_edit)) {print "<br> Inserted in DataBase sppindex !";} // else {die ('Error in query: ' . $sql_edit . mysql_error());} and nothing had improved... Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1115960 Share on other sites More sharing options...
rwwd Posted September 26, 2010 Share Posted September 26, 2010 Have you tried any other browser at all, just to see if it M$ being it usual clunky self. And yes, sleep(); is rather good function, just as important as empty() and isset() IMO Rw Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1115967 Share on other sites More sharing options...
marz Posted September 26, 2010 Author Share Posted September 26, 2010 Update 2: I have comented out numerous print commands so as about 80% of the printing in the browser window is eliminated. I have seen improvemnet, since I reached the 4000 files in 2 minutes, but then things slowed down and eventually became dead arounf 4700 files. I noticed that if I click on the window of another application (eg: Outlook express) and then back on the broser window, the script processes a chunk and freezes again. I cant make any sense in my limited knowledge :-( Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1115970 Share on other sites More sharing options...
marz Posted September 26, 2010 Author Share Posted September 26, 2010 I only have IE7 Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1115971 Share on other sites More sharing options...
marz Posted September 26, 2010 Author Share Posted September 26, 2010 Quote Are you running this from your browser? Don't. Run it from command line. HOW? Will the print text show up or it is meant that the script executes in a kind of silent mode? Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1115980 Share on other sites More sharing options...
marz Posted September 26, 2010 Author Share Posted September 26, 2010 Quote IT IS VITALLY IMPORTANT that when using opendir and then scanning with readdir, that you test opendir to see if it has returned a valid pointer. The following code does not : <?php $dir = opendir($mydir); while((false!==($file=readdir($dir)))) if($file!="." and $file !="..") unlink($mydir.'/'.$file); closedir($dir); ?> if $mydir is not a valid directory resource (for example, the directory in question has been deleted), then the code will loop infinitely. Of course, it'll stop eventually if you have set a max execution time, but in the meantime it will visibly slow the server. Very true, I used to have this problem in the past and finally realised that if a dir does not exist, it will loop for infinity and hence freezes. However that gives a kind of error code in the errorlog.txt and this is not the issue becasue I have checked taht all dirs are there. What else can it be???? Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1115982 Share on other sites More sharing options...
rwwd Posted September 26, 2010 Share Posted September 26, 2010 Get something else to try then, safari. opera, firefox, chrome, these should be standard browsers installed on your machine if you are developing, this then tests code to see if you are being displayed as you intended in every browser. IE is RENOWNED for being a really clunky and very slow, and not showing lots of css/div layouts properly. Rw Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1115984 Share on other sites More sharing options...
marz Posted September 26, 2010 Author Share Posted September 26, 2010 Here is another part of the puzzle. I let the script to run with exec_time of 5 mins (300seconds) and, as expected, the script failed with the exe limit time exceeded error after about 700/1000 scanned directories. I expected that iexplorer will become back to normal then, but no, it was still buggy, slow and had to close it with taskmanager. ?? Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1115996 Share on other sites More sharing options...
marz Posted September 26, 2010 Author Share Posted September 26, 2010 Is there a way to flush the memory or in old basic language - "Clear the screen" of iexplorer with php. Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1116001 Share on other sites More sharing options...
roopurt18 Posted September 26, 2010 Share Posted September 26, 2010 Clear the screen? No. If the browser is locking up so bad that you're having to close it with task manager then I'm 95% sure the problem is too much output to the browser. Do as suggested and run from a command line: Start -> run -> cmd cd \directory\with\php\file\in\it php scriptname.php If you want you can redirect the output to a file: Start -> run -> cmd cd \directory\with\php\file\in\it php scriptname.php > scriptname.html And then you can see the file size of scriptname.html. If its multiple megabytes then too much output is your problem. Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1116020 Share on other sites More sharing options...
marz Posted September 27, 2010 Author Share Posted September 27, 2010 I tried the command line but I got an error. Probably it is finding some libraries in thge old installation dir (F:\INTERNET\Coding\PHP5\). How can I change so as it searches in the actual dir that php is installed (K:/server/php). DO I have some old php.ini around? K:\Documents and Settings\Stephen>K:/server/php/php PHP Warning: PHP Startup: gd: Unable to initialize module Module compiled with module API=20060613 PHP compiled with module API=20090626 These options need to match in Unknown on line 0 PHP Warning: PHP Startup: Unable to load dynamic library 'F:\INTERNET\Coding\PH P5\ext\php_mysql.dll' - Invalid access to memory location. in Unknown on line 0 PHP Warning: PHP Startup: Unable to load dynamic library 'F:\INTERNET\Coding\PH P5\ext\php_mysqli.dll' - The specified procedure could not be found. in Unknown on line 0 PHP Warning: Directive 'magic_quotes_gpc' is deprecated in PHP 5.3 and greater in Unknown on line 0 Warning: Directive 'magic_quotes_gpc' is deprecated in PHP 5.3 and greater in Un known on line 0 PHP Warning: Directive 'magic_quotes_runtime' is deprecated in PHP 5.3 and grea ter in Unknown on line 0 Warning: Directive 'magic_quotes_runtime' is deprecated in PHP 5.3 and greater i n Unknown on line 0 ^A Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1116282 Share on other sites More sharing options...
marz Posted September 27, 2010 Author Share Posted September 27, 2010 I have fixed this by setting the path pointing to the correct dor from the Environment variables settings. The variable is called PHPRC. I test the script from the command line execution and keep you posted Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1116335 Share on other sites More sharing options...
marz Posted September 27, 2010 Author Share Posted September 27, 2010 HALF SUCCESS I managed to run the script from CMD line which took less than 2 mins. Then I did a step further, I created a short cut link which also worked well, except one thing which I like to have functioning. I failed to create a script output file as instructed in the 2 previous posts. I used the following command parse: K:\server\php\php.exe g:\marz-kreations\Wildplants\!php\Image_tools.php > G:\MarZ-Kreations\WildPlants\!php\exec\Image_Tools.html I created an exec dir but no html file was created. When running the command (via double clicking the link) the script ran in the black cmd window Quote Link to comment https://forums.phpfreaks.com/topic/214016-execution-of-a-php-code-takes-too-long/#findComment-1116343 Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.