Calling pages and close connection after x second

[FaithRaven]

Hello

 

I created a crawler that parse huge amounts of XML files and insert/update data in a database. I have uploaded this crawler on multiple servers and I want to call all of them from the main project.

 

In the main project I have something like:

file_get_contents("http://crawler1.net");

file_get_contents("http://crawler2.net");

file_get_contents("http://crawler3.net");

 

It calls crawler1, wait until it finish running, and then call crawler2.

 

What I want to achieve is to call crawler1, and then call crawler2 right away without waiting for crawler1 to finish loading.

 

I do not need anything from what a crawler outputs, they all work independently and insert/update data in the same database. All I need is to call them so they start crawling.

[trq]

You might try...

 

<?php

  exec('/usr/bin/wget http://crawler1.net &');
  exec('/usr/bin/wget http://crawler2.net &');
  exec('/usr/bin/wget http://crawler3.net &');

?>

[FaithRaven]

Wouldn't that create 3 processes on the server ?

 

My problem started from the fact that my host allows me to have only 25 processes. Currently the script calling the crawlers blocks and keeps it's process occupied.

[Orio]

I never used it before, but PHP offers a library that allows process forking called PCNTL.

 

http://www.google.com/search?hl=en&q=php+forking

http://www.php.net/pcntl

 

Orio.

[trq]

Wouldn't that create 3 processes on the server ?

 

Yes, but there is no way of doing what you want without creating extra processes. That how it will work.