Basic Question about Threading and PHP...

[whit3fir3]

I have a page that I am working on and it is taking several hours to process.  The basics of what the page does is get all the items out of a database then with cURL download some HTML parse through it and if anything has changes update values back in the database.  90% of the time is spent waiting on libcurl to download the page and the page is VERY low on CPU usage.  In the effort of trying to speed things up I thought about trying to thread the download process, but from a quick google search it appears that PHP lacks Threading support.  Does anyone know of anything that could be done to speed things up?  Maybe something I missed with cURL that would make it download faster or an alternative to Threading?

 

Thanks,

 

Whit3fir3

[Deoctor]

can u paste ur code so that it would be more understandable..

[whit3fir3]

There are large sections of code that will be filled in with SUDO code as I am still developing the page, but this should give you an idea as to what is going on here:

 

$sql = 'SELECT id, name, URL, current_price from Items ORDER BY id DESC';
$result = mysql_query($sql) or die("Query failed : " . mysql_error());

while ($line = mysql_fetch_assoc($result))
{
$value[] = $line;
}
//THE PREVIOUS CODE RETURNS AN ARRAY NAMED $value WITH ALL ITEMS FROM DB IN IT

$item = array();
foreach($value as $row){
$bprice = array();
if(getPrice($row['URL'], $bprice)){
	if($row['current_price'] != $bprice['max']){
			array_push($item, $row);
	}
}else{
	$error = "Error Message about not being able to process the request for the item...?";
}
}

//HERE ARE THE IMPORTANT PARTS OF THE getPrice Function

function getPrice($URL, &$newPrice){
$ch = curl_init();
	if($ch){		
		$curlOpts = array(CURLOPT_URL => $loginURL,
					CURLOPT_USERAGENT => $agent,
					CURLOPT_RETURNTRANSFER => 1,
					CURLOPT_COOKIEJAR => $cookie,
					CURLOPT_COOKIEFILE => $cookie,
					CURLOPT_SSL_VERIFYPEER => false,
					CURLOPT_SSL_VERIFYHOST => false,
					CURLOPT_POST => true,
					CURLOPT_POSTFIELDS => 'REMOVED DUE TO USERNAME & PASSWORD');
curl_setopt_array($ch, $curlOpts);
curl_exec ($ch);
curl_setopt($ch, CURLOPT_URL,$URL);
curl_setopt($ch, CURLOPT_POSTFIELDS , 'POST FIELDS REMOVED');
$priceInfo = curl_exec($ch);
if($priceInfo){
//SEARCH HTML CODE FOR CERTAIN STRINGS
$newPrice['max'] = SOME STRING FROM THE HTML;
}else{
$returnValue = false;
}
return $returnValue;
}

 

In the getPrice function I am currently running I have lots more checking on the cURL process to test for anything failing and then the HTML parsing code.  If you have any questions about something in the code that I have posted please let me know.

[roopurt18]

I'm not sure what kind of threading support PHP has on Windows, but you can fork() in Linux.

 

Anyways, the best thing to do in your case is:

 

1) User submits form

2) Page saves a flag in the database, responds to user that request is pending

3) A cron job or scheduled task checks for the database flag, if the flag is set the script is run

4) Notify the user via e-mail when the job is finished.

[whit3fir3]

Correct me is I am wrong, but you can only FORK from a PHP script and not from a PHP web page?  If so that would explain the setting a value in the DB and then a cron job to check said value and to execute if necessary.

[trq]

Correct me is I am wrong, but you can only FORK from a PHP script and not from a PHP web page?  If so that would explain the setting a value in the DB and then a cron job to check said value and to execute if necessary.

 

That is correct. There is pcntl_fork but it is nopt recommended for use within a server environment.