jos. Posted December 14, 2007 Share Posted December 14, 2007 Didn,t want to put this with conversations about cats and dogs in the Misc... section... I have a number of addon domains and need to know If I place a robots.txt file in my main directory to not associate my site with the addon sites, will that prevent a spider from visiting those sites because the spider would be heading to the main dir of that IP first? Or does it know to hit the IP but start at the sub dir? Thanks for any help in advance. Jos. Quote Link to comment Share on other sites More sharing options...
trq Posted December 16, 2007 Share Posted December 16, 2007 How are your domains defined? They ought to be considered as completely seperate sites. Quote Link to comment Share on other sites More sharing options...
jos. Posted December 17, 2007 Author Share Posted December 17, 2007 They are defined as separate sites but I googled my domain name the other day and got some results containing my subdirectories and their content. So, simple fix right? Robots.txt in my main directory and block those. But my concerns are as stated above. Thanks for your help. Jos. Quote Link to comment Share on other sites More sharing options...
trq Posted December 17, 2007 Share Posted December 17, 2007 If they are seperate sites, why are they within the same document root? Quote Link to comment Share on other sites More sharing options...
jos. Posted December 17, 2007 Author Share Posted December 17, 2007 Because that is how my host employs addon domains. the hierarchy is as such: public_html/ ---public_html/addonDirectory ---public_html/addonDirectory/subDirectories so if you type: http://www.mysite.com/ you land on public_html/index.php ('A') and if you were to type: http://www.myaddonDomainName.com/ you land on public_html/addonDomainName/index.php ('B') so I want to know if a spider is deployed to http://www.myaddonDomainName.com if he will start at ('A') or ('B') Because they both have the same IP address. Thank you for your patience! Jos. Quote Link to comment Share on other sites More sharing options...
creative182002 Posted December 3, 2008 Share Posted December 3, 2008 Yes the Spider will crawl all the links related to the directories and sub directories. What you want is that the spider must not follow the sub sites from you main website. The simple solution for this is robots.txt file which you have already implemented. I think robots.txt file can help you in resolving this issue very easily Quote Link to comment Share on other sites More sharing options...
forumGuy201 Posted March 16, 2010 Share Posted March 16, 2010 The spider will follow your links in the format you provided (A & B) you can however instead of using robot.txt use this meta tag at the top of your page. <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> Quote Link to comment Share on other sites More sharing options...
Maq Posted March 16, 2010 Share Posted March 16, 2010 Don't resurrect ancient threads. Last reply: December 03, 2008, 01:58:35 AM. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.