Jump to content

Robots.txt


liamloveslearning

Recommended Posts

Hi everyone, I have my robots.txt file in my root foler, my robots.txt is

 

User-agent: *
Allow: /
User-agent: *
Disallow: /eshot/
User-agent: *
Disallow: /newsletters/
User-agent: *
Disallow: /Connections/
User-agent: *
Disallow: /libraries/
User-agent: *
Disallow: /resources/
User-agent: *
Disallow: /sifr/
User-agent: *
Disallow: /SpryAssets/

 

in webmaster tools on google I receive the message "www.mywebiste.com Allowed by line 2: Allow: /

Detected as a directory; specific files may have different restrictions"

 

Does this mean where I have my allow all, that its contradicting my disallow restrictions?

Link to comment
https://forums.phpfreaks.com/topic/188448-robotstxt/
Share on other sites

I took the freedom to edit your robots.txt file,

 

try this :

User-agent: *
Disallow: /eshot/
Disallow: /newsletters/
Disallow: /Connections/
Disallow: /libraries/
Disallow: /resources/
Disallow: /sifr/
Disallow: /SpryAssets/

 

That's basically is going to do all you're looking for, it will disallow those directories in all search engines,

Link to comment
https://forums.phpfreaks.com/topic/188448-robotstxt/#findComment-995054
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.