Using Curl_multi for processing multiple URLs

[imperium2335]

Hi,

 

I am at a loss as to how to implement this. I would like to be able to automatically assign a number of urls taken from a file to x amount of curl_multi handles determined by the user on the form, so if i had 100 urls, and the user types 2 threads, each handle will process and go through 50 urls each.

 

The problems I face are how do I get php to create the number of handles based on the users' preference, are variable variables the only way? Because I really don't want to go there as my head will explode.

 

Also, how do I ensure that thread A can not process urls that Thread B is currently working on? Could this be done by popping the url off an array just before the thread goes to work on it?

 

Many thanks, Tom.

[Daniel0]

That's not how the curl_multi_* stuff works. If you want to create multi threaded scripts, have a look at the pcntl functions.

[imperium2335]

wow, that looks like a lot of reading! Thanks I will check it out.

[imperium2335]

So whats the point of curl_multi if it cant process multiple urls and drop the contents into a string? Is there no way of doing that without learning this new branch of php?

[Daniel0]

It is supposed to process multiple URLs in parallel, but not in the way you're saying.

 

Say you have your URLs in an array called $urls. Then you can do this:

 

<?php
$mh = curl_init_multi();

$handles = array();
foreach ($urls as $i => $url) {
$handles[$i] = curl_init($url);
curl_setopt($handles[$i], CURLOPT_HEADER, false);
curl_setopt($handles[$i], CURLOPT_RETURNTRANSFER, true);

curl_multi_add_handle($mh, $handles[$i]);
}

do {
curl_multi_exec($mh, $running);
} while ($running > 0);

// get content using curl_multi_getcontent() here

foreach ($handles as $handle) {
curl_multi_remove_handle($mh, $handle);
curl_close($handle);
}

curl_multi_close($mh);