Jump to content

Archived

This topic is now archived and is closed to further replies.

Anidazen

Parsing many sites similtaneously.

Recommended Posts

Hello.

I've got a script that involves parsing 10 seperate websites. Some of these have quite slow servers at times. That's a loading time of 20 seconds odd for my script, almost entirely from these sites.

Is there any way at all to CURL all 10 at once, or as many similtaneously as possible, to minimise this waiting? Or is that one of the few web-scripting tasks that is truly beyond the reach of PHP?


Please advise!

Share this post


Link to post
Share on other sites
I would recommend doing the processing of the sites in the background. Here is a tutorial http://www.phpfreaks.com/tutorials/71/0.php. This says that it is forking but it realy isn't. It is realy just a tutorial on how to use an exec function to run a process in the background.

Hope this helps,
Tom

Share this post


Link to post
Share on other sites
I've heard about something called CURL_multi, but I haven't been able to find a single piece of relevant documentation or a tutorial or anything. Anyone able to shed some light? (I couldn't even find some sample source to work it out myself from!)

Share this post


Link to post
Share on other sites

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.