Script Stopping at what (seems) to be random.

[jscix]

I've got a spidering script, that indexes a websites contents.. it works beautifully on smaller sites, however when crawling large sites that take a good amount of time.. the script seems to just randomly stop. I've actually seen it complete a full crawl of a large site, but 95% (estimate) of the time, it will stop indexing and just quit.

 

I have already added,

set_time_limit(3600);
ini_set('max_execution_time', 3600);

 

I don't know what else could be causing it. If anyone has some suggestions, I'd be open to hear them.

 

Thanks