daydreamer Posted October 24, 2009 Share Posted October 24, 2009 Hi. I have written a web crawler script. It will visit a large number of URL's with cURL. After around 2-3 minutes of running, it will just stop, with no error output or notices. I have these settings: set_time_limit(0); ini_set('display_errors',1); error_reporting(E_ALL|E_STRICT); Any ideas why it would just stop? Link to comment https://forums.phpfreaks.com/topic/178877-solved-why-does-my-crawler-script-suddenly-end-with-no-error/ Share on other sites More sharing options...
Mark Baker Posted October 24, 2009 Share Posted October 24, 2009 Are you running this from a web page? or as a command line script? Link to comment https://forums.phpfreaks.com/topic/178877-solved-why-does-my-crawler-script-suddenly-end-with-no-error/#findComment-943698 Share on other sites More sharing options...
daydreamer Posted October 24, 2009 Author Share Posted October 24, 2009 Yes im running it from Firefox. Link to comment https://forums.phpfreaks.com/topic/178877-solved-why-does-my-crawler-script-suddenly-end-with-no-error/#findComment-943705 Share on other sites More sharing options...
Mark Baker Posted October 24, 2009 Share Posted October 24, 2009 Then your web server is probably timing out Link to comment https://forums.phpfreaks.com/topic/178877-solved-why-does-my-crawler-script-suddenly-end-with-no-error/#findComment-943715 Share on other sites More sharing options...
daydreamer Posted October 24, 2009 Author Share Posted October 24, 2009 Tried running it from the windows command line no luck. Cannot find any apache settings for timeouts. Link to comment https://forums.phpfreaks.com/topic/178877-solved-why-does-my-crawler-script-suddenly-end-with-no-error/#findComment-943729 Share on other sites More sharing options...
Mark Baker Posted October 25, 2009 Share Posted October 25, 2009 If it's a long running task, it should almost certainly be run from the command line rather than from a we browser. However, you do need to check the php.ini settings for the command line, it's quite possible that it's running against a different php.ini file. Running a phpinfo() script from the command line should provide that information Link to comment https://forums.phpfreaks.com/topic/178877-solved-why-does-my-crawler-script-suddenly-end-with-no-error/#findComment-943961 Share on other sites More sharing options...
daydreamer Posted October 25, 2009 Author Share Posted October 25, 2009 Thanks, but it was a coding error. I put an exit in my code to stop it at 20 loops to prevent me accidentally running the full script, but then I forgot to remove it. It is an Awesome feature... Link to comment https://forums.phpfreaks.com/topic/178877-solved-why-does-my-crawler-script-suddenly-end-with-no-error/#findComment-944329 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.