Is it Possible or just a fiction?

[natasha_thomas]

Folks,

 

I just had a thought might sound dumb but will be dumber if i dnt share.

 

Is it possible to ask googlebots to spider/crawl my site slowly instad of sending multiple threads to crawl my site.

 

By slowly i do not mean just spider one of two pages, what i mean is spider one page then another and another, so i can better manage my server resources.

 

I think there is something crawl delay setting or something like that can be done in robots.txt

 

All i am worried is, will it work against the SEO of My sites?

 

Any thoughts?

 

Cheers

NT

 

 

[Adam]

If you sign-up for Google's web master tools you can set a custom crawl rate within Site Configuration > Settings. I'm not sure through a robots.txt file though..

[dreamwest]

User-agent: googlebot

Request-rate: 1/10  # maximum rate is one page every 10 seconds.

[natasha_thomas]

User-agent: googlebot

Request-rate: 1/10  # maximum rate is one page every 10 seconds.

 

Thanks DW.

 

1) Will it effect my SEO with Googlebots negatively, as i am restricting them to crawl slow?

 

2) Do i need to register to Webmaster tool for this to work? (Because i do not want to register my site with Google Webmaster)

 

Cheers

N

[dreamwest]

User-agent: googlebot

Request-rate: 1/10  # maximum rate is one page every 10 seconds.

 

Thanks DW.

 

1) Will it effect my SEO with Googlebots negatively, as i am restricting them to crawl slow?

 

2) Do i need to register to Webmaster tool for this to work? (Because i do not want to register my site with Google Webmaster)

 

Cheers

N

 

Just throw it in robots.txt and your done