[SOLVED] Why does my crawler script suddenly end with no error?

[daydreamer]

Hi.

 

I have written a web crawler script. It will visit a large number of URL's with cURL.

 

After around 2-3 minutes of running, it will just stop, with no error output or notices.

 

I have these settings:

set_time_limit(0);
ini_set('display_errors',1);
error_reporting(E_ALL|E_STRICT);

 

Any ideas why it would just stop?

[Mark Baker]

Are you running this from a web page? or as a command line script?

[daydreamer]

Yes im running it from Firefox.

[Mark Baker]

Then your web server is probably timing out

[daydreamer]

Tried running it from the windows command line no luck.

 

Cannot find any apache settings for timeouts.

[Mark Baker]

If it's a long running task, it should almost certainly be run from the command line rather than from a we browser.

However, you do need to check the php.ini settings for the command line, it's quite possible that it's running against a different php.ini file. Running a phpinfo() script from the command line should provide that information

[daydreamer]

Thanks, but it was a coding error.

 

I put an exit in my code to stop it at 20 loops to prevent me accidentally running the full script, but then I forgot to remove it.

 

It is an Awesome feature...