natasha_thomas Posted May 20, 2011 Share Posted May 20, 2011 Folks, I just had a thought might sound dumb but will be dumber if i dnt share. Is it possible to ask googlebots to spider/crawl my site slowly instad of sending multiple threads to crawl my site. By slowly i do not mean just spider one of two pages, what i mean is spider one page then another and another, so i can better manage my server resources. I think there is something crawl delay setting or something like that can be done in robots.txt All i am worried is, will it work against the SEO of My sites? Any thoughts? Cheers NT Quote Link to comment Share on other sites More sharing options...
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.