natasha_thomas Posted May 20, 2011 Share Posted May 20, 2011 Folks, I just had a thought might sound dumb but will be dumber if i dnt share. Is it possible to ask googlebots to spider/crawl my site slowly instad of sending multiple threads to crawl my site. By slowly i do not mean just spider one of two pages, what i mean is spider one page then another and another, so i can better manage my server resources. I think there is something crawl delay setting or something like that can be done in robots.txt All i am worried is, will it work against the SEO of My sites? Any thoughts? Cheers NT Link to comment https://forums.phpfreaks.com/topic/236961-is-it-possible-or-just-a-fiction/ Share on other sites More sharing options...
Adam Posted May 20, 2011 Share Posted May 20, 2011 If you sign-up for Google's web master tools you can set a custom crawl rate within Site Configuration > Settings. I'm not sure through a robots.txt file though.. Link to comment https://forums.phpfreaks.com/topic/236961-is-it-possible-or-just-a-fiction/#findComment-1218045 Share on other sites More sharing options...
dreamwest Posted May 20, 2011 Share Posted May 20, 2011 User-agent: googlebot Request-rate: 1/10 # maximum rate is one page every 10 seconds. Link to comment https://forums.phpfreaks.com/topic/236961-is-it-possible-or-just-a-fiction/#findComment-1218246 Share on other sites More sharing options...
natasha_thomas Posted May 21, 2011 Author Share Posted May 21, 2011 User-agent: googlebot Request-rate: 1/10 # maximum rate is one page every 10 seconds. Thanks DW. 1) Will it effect my SEO with Googlebots negatively, as i am restricting them to crawl slow? 2) Do i need to register to Webmaster tool for this to work? (Because i do not want to register my site with Google Webmaster) Cheers N Link to comment https://forums.phpfreaks.com/topic/236961-is-it-possible-or-just-a-fiction/#findComment-1218454 Share on other sites More sharing options...
dreamwest Posted May 22, 2011 Share Posted May 22, 2011 User-agent: googlebot Request-rate: 1/10 # maximum rate is one page every 10 seconds. Thanks DW. 1) Will it effect my SEO with Googlebots negatively, as i am restricting them to crawl slow? 2) Do i need to register to Webmaster tool for this to work? (Because i do not want to register my site with Google Webmaster) Cheers N Just throw it in robots.txt and your done Link to comment https://forums.phpfreaks.com/topic/236961-is-it-possible-or-just-a-fiction/#findComment-1218601 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.