natasha_thomas Posted May 20, 2011 Share Posted May 20, 2011 Folks, I just had a thought might sound dumb but will be dumber if i dnt share. Is it possible to ask googlebots to spider/crawl my site slowly instad of sending multiple threads to crawl my site. By slowly i do not mean just spider one of two pages, what i mean is spider one page then another and another, so i can better manage my server resources. I think there is something crawl delay setting or something like that can be done in robots.txt All i am worried is, will it work against the SEO of My sites? Any thoughts? Cheers NT Quote Link to comment https://forums.phpfreaks.com/topic/236961-is-it-possible-or-just-a-fiction/ Share on other sites More sharing options...
Adam Posted May 20, 2011 Share Posted May 20, 2011 If you sign-up for Google's web master tools you can set a custom crawl rate within Site Configuration > Settings. I'm not sure through a robots.txt file though.. Quote Link to comment https://forums.phpfreaks.com/topic/236961-is-it-possible-or-just-a-fiction/#findComment-1218045 Share on other sites More sharing options...
dreamwest Posted May 20, 2011 Share Posted May 20, 2011 User-agent: googlebot Request-rate: 1/10 # maximum rate is one page every 10 seconds. Quote Link to comment https://forums.phpfreaks.com/topic/236961-is-it-possible-or-just-a-fiction/#findComment-1218246 Share on other sites More sharing options...
natasha_thomas Posted May 21, 2011 Author Share Posted May 21, 2011 User-agent: googlebot Request-rate: 1/10 # maximum rate is one page every 10 seconds. Thanks DW. 1) Will it effect my SEO with Googlebots negatively, as i am restricting them to crawl slow? 2) Do i need to register to Webmaster tool for this to work? (Because i do not want to register my site with Google Webmaster) Cheers N Quote Link to comment https://forums.phpfreaks.com/topic/236961-is-it-possible-or-just-a-fiction/#findComment-1218454 Share on other sites More sharing options...
dreamwest Posted May 22, 2011 Share Posted May 22, 2011 User-agent: googlebot Request-rate: 1/10 # maximum rate is one page every 10 seconds. Thanks DW. 1) Will it effect my SEO with Googlebots negatively, as i am restricting them to crawl slow? 2) Do i need to register to Webmaster tool for this to work? (Because i do not want to register my site with Google Webmaster) Cheers N Just throw it in robots.txt and your done Quote Link to comment https://forums.phpfreaks.com/topic/236961-is-it-possible-or-just-a-fiction/#findComment-1218601 Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.