vozzek Posted February 28, 2011 Share Posted February 28, 2011 Hi all, Okay, I have a domain, let's call it domain1.com with another Add-on domain linked to a folder within the main url (let's call it domain2.com). If I look at domain2 in google's webmaster tools, everything looks fine. If I look at domain1 in google's websmaster tools, it sees the entire domain2 folder as part of domain1. Therefore, it sees all kinds of "broken" links. My question is this: if I disallow that folder in the robots.txt file for domain1, will it disable google (and other web spiders) from crawling domain2 entirely? For example, if my robots.txt file looks like this: User-agent: * Disallow: /style.css Disallow: /domain2.com will it create problems with google being able to see my add-on domain? Thanks in advance for the help. Link to comment https://forums.phpfreaks.com/topic/229161-robotstxt-question/ Share on other sites More sharing options...
haku Posted March 1, 2011 Share Posted March 1, 2011 I believe you can check your robots.txt in webmaster tools - so make the changes and then check both domains in webmaster toos, and see what it tells you. Link to comment https://forums.phpfreaks.com/topic/229161-robotstxt-question/#findComment-1181249 Share on other sites More sharing options...
Web Developer Posted March 7, 2011 Share Posted March 7, 2011 Hi all, Okay, I have a domain, let's call it domain1.com with another Add-on domain linked to a folder within the main url (let's call it domain2.com). If I look at domain2 in google's webmaster tools, everything looks fine. If I look at domain1 in google's websmaster tools, it sees the entire domain2 folder as part of domain1. Therefore, it sees all kinds of "broken" links. My question is this: if I disallow that folder in the robots.txt file for domain1, will it disable google (and other web spiders) from crawling domain2 entirely? For example, if my robots.txt file looks like this: User-agent: * Disallow: /style.css Disallow: /domain2.com will it create problems with google being able to see my add-on domain? Thanks in advance for the help. Yes, if you disable domail2.com in robots.txt as you mentioned above, no crawler will crawl domain2.com while visiting domain1.com Link to comment https://forums.phpfreaks.com/topic/229161-robotstxt-question/#findComment-1183934 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.