Curl Multi

[Anidazen]

Hi there.

 

 

I am very interested in curl_multi, but have been unable to find much information on it at all. For something so tricky and powerful, it seems to be quite badly documented. (Or I suck at Google. One of the two.) Does anyone know a good article on curl_multi?

 

 

The only decent example I managed to find was posted as a comment in one page of the PHP.net manual. This would appear to fetch three urls, and seems to have scope to be easily scaled. However, there's some real issues I have here, and I need help understanding them.

 

I assume in the below code that at the end you can access these pages as $res[$i]. But to put this to practical use I would really like to know if this could be used to process pages *as they come in*. I run a specialist site using spiders in real-time to fetch info, and I need to show the first results as soon as they come in. I can't wait for all of them. Is there a way to process info as each individual handle is ready? Also - Is there a way to identify the URL of the site, as well as the content?

 

 

 

 

 

Here's the example code I'm quoting. (I assume the print_r at the end is an error, and should be $res[$i] instead.)

 

 

 

 

<?php 
$connomains = array( 
   "http://www.cnn.com/", 
   "http://www.canada.com/", 
   "http://www.yahoo.com/" 
); 

$mh = curl_multi_init(); 

foreach ($connomains as $i => $url) { 
  $conn[$i] = curl_init($url); 
  curl_setopt($conn[$i], CURLOPT_RETURNTRANSFER, 1); 
  curl_multi_add_handle ($mh,$conn[$i]); 
} 

// start performing the request 
do { 
  $mrc = curl_multi_exec($mh, $active); 
} while ($mrc == CURLM_CALL_MULTI_PERFORM); 

while ($active and $mrc == CURLM_OK) { 
  // wait for network 
  if (curl_multi_select($mh) != -1) { 
    // pull in any new data, or at least handle timeouts 
    do { 
      $mrc = curl_multi_exec($mh, $active); 
    } while ($mrc == CURLM_CALL_MULTI_PERFORM); 
  } 
} 

if ($mrc != CURLM_OK) { 
  print "Curl multi read error $mrc\n"; 
} 

// retrieve data 
foreach ($connomains as $i => $url) { 
  if (($err = curl_error($conn[$i])) == '') { 
    $res[$i]=curl_multi_getcontent($conn[$i]); 
  } else { 
    print "Curl error on handle $i: $err\n"; 
  } 
  curl_multi_remove_handle($mh,$conn[$i]); 
  curl_close($conn[$i]); 
} 
curl_multi_close($mh); 

print_r($res); 
?>