whit3fir3 Posted December 21, 2009 Share Posted December 21, 2009 I have a page that I am working on and it is taking several hours to process. The basics of what the page does is get all the items out of a database then with cURL download some HTML parse through it and if anything has changes update values back in the database. 90% of the time is spent waiting on libcurl to download the page and the page is VERY low on CPU usage. In the effort of trying to speed things up I thought about trying to thread the download process, but from a quick google search it appears that PHP lacks Threading support. Does anyone know of anything that could be done to speed things up? Maybe something I missed with cURL that would make it download faster or an alternative to Threading? Thanks, Whit3fir3 Quote Link to comment Share on other sites More sharing options...
Deoctor Posted December 21, 2009 Share Posted December 21, 2009 can u paste ur code so that it would be more understandable.. Quote Link to comment Share on other sites More sharing options...
whit3fir3 Posted December 21, 2009 Author Share Posted December 21, 2009 There are large sections of code that will be filled in with SUDO code as I am still developing the page, but this should give you an idea as to what is going on here: $sql = 'SELECT id, name, URL, current_price from Items ORDER BY id DESC'; $result = mysql_query($sql) or die("Query failed : " . mysql_error()); while ($line = mysql_fetch_assoc($result)) { $value[] = $line; } //THE PREVIOUS CODE RETURNS AN ARRAY NAMED $value WITH ALL ITEMS FROM DB IN IT $item = array(); foreach($value as $row){ $bprice = array(); if(getPrice($row['URL'], $bprice)){ if($row['current_price'] != $bprice['max']){ array_push($item, $row); } }else{ $error = "Error Message about not being able to process the request for the item...?"; } } //HERE ARE THE IMPORTANT PARTS OF THE getPrice Function function getPrice($URL, &$newPrice){ $ch = curl_init(); if($ch){ $curlOpts = array(CURLOPT_URL => $loginURL, CURLOPT_USERAGENT => $agent, CURLOPT_RETURNTRANSFER => 1, CURLOPT_COOKIEJAR => $cookie, CURLOPT_COOKIEFILE => $cookie, CURLOPT_SSL_VERIFYPEER => false, CURLOPT_SSL_VERIFYHOST => false, CURLOPT_POST => true, CURLOPT_POSTFIELDS => 'REMOVED DUE TO USERNAME & PASSWORD'); curl_setopt_array($ch, $curlOpts); curl_exec ($ch); curl_setopt($ch, CURLOPT_URL,$URL); curl_setopt($ch, CURLOPT_POSTFIELDS , 'POST FIELDS REMOVED'); $priceInfo = curl_exec($ch); if($priceInfo){ //SEARCH HTML CODE FOR CERTAIN STRINGS $newPrice['max'] = SOME STRING FROM THE HTML; }else{ $returnValue = false; } return $returnValue; } In the getPrice function I am currently running I have lots more checking on the cURL process to test for anything failing and then the HTML parsing code. If you have any questions about something in the code that I have posted please let me know. Quote Link to comment Share on other sites More sharing options...
roopurt18 Posted December 21, 2009 Share Posted December 21, 2009 I'm not sure what kind of threading support PHP has on Windows, but you can fork() in Linux. Anyways, the best thing to do in your case is: 1) User submits form 2) Page saves a flag in the database, responds to user that request is pending 3) A cron job or scheduled task checks for the database flag, if the flag is set the script is run 4) Notify the user via e-mail when the job is finished. Quote Link to comment Share on other sites More sharing options...
whit3fir3 Posted December 22, 2009 Author Share Posted December 22, 2009 Correct me is I am wrong, but you can only FORK from a PHP script and not from a PHP web page? If so that would explain the setting a value in the DB and then a cron job to check said value and to execute if necessary. Quote Link to comment Share on other sites More sharing options...
trq Posted December 22, 2009 Share Posted December 22, 2009 Correct me is I am wrong, but you can only FORK from a PHP script and not from a PHP web page? If so that would explain the setting a value in the DB and then a cron job to check said value and to execute if necessary. That is correct. There is pcntl_fork but it is nopt recommended for use within a server environment. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.