jscix Posted February 18, 2008 Share Posted February 18, 2008 I've got a spidering script, that indexes a websites contents.. it works beautifully on smaller sites, however when crawling large sites that take a good amount of time.. the script seems to just randomly stop. I've actually seen it complete a full crawl of a large site, but 95% (estimate) of the time, it will stop indexing and just quit. I have already added, set_time_limit(3600); ini_set('max_execution_time', 3600); I don't know what else could be causing it. If anyone has some suggestions, I'd be open to hear them. Thanks Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.