Jump to content

Is it Possible or just a fiction?


natasha_thomas

Recommended Posts

Folks,

 

I just had a thought might sound dumb but will be dumber if i dnt share.

 

Is it possible to ask googlebots to spider/crawl my site slowly instad of sending multiple threads to crawl my site.

 

By slowly i do not mean just spider one of two pages, what i mean is spider one page then another and another, so i can better manage my server resources.

 

I think there is something crawl delay setting or something like that can be done in robots.txt

 

All i am worried is, will it work against the SEO of My sites?

 

Any thoughts?

 

Cheers

NT

 

 

Link to comment
Share on other sites

User-agent: googlebot

Request-rate: 1/10  # maximum rate is one page every 10 seconds.

 

Thanks DW.

 

1) Will it effect my SEO with Googlebots negatively, as i am restricting them to crawl slow?

 

2) Do i need to register to Webmaster tool for this to work? (Because i do not want to register my site with Google Webmaster)

 

Cheers

N

Link to comment
Share on other sites

User-agent: googlebot

Request-rate: 1/10  # maximum rate is one page every 10 seconds.

 

Thanks DW.

 

1) Will it effect my SEO with Googlebots negatively, as i am restricting them to crawl slow?

 

2) Do i need to register to Webmaster tool for this to work? (Because i do not want to register my site with Google Webmaster)

 

Cheers

N

 

Just throw it in robots.txt and your done

Link to comment
Share on other sites

This thread is more than a year old. Please don't revive it unless you have something important to add.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.