liamloveslearning Posted January 12, 2010 Share Posted January 12, 2010 Hi everyone, Im just after some advice regarding my robots.txt file, I have to directories on my server, httpdocs and httpsdocs, httpsdocs contains my webstore. I want to stop search engines crawling my store as there are a lot of missing oages etc and there not very 'SEO friendly URLS', ive googled around the subject and it seems I need to include User-agent: * Disallow: /httpsdocs/ in my robots.txt, if I was to do this and place the robots.txt above http and httpsdocs on my server shall it theoretically work? Thanks Link to comment https://forums.phpfreaks.com/topic/188189-robotstxt/ Share on other sites More sharing options...
cags Posted January 12, 2010 Share Posted January 12, 2010 Yes, that should work for any bot that follows the rules (ie search engine bots which is what your targeting). Just add that into a file and drop it in your root directory. Link to comment https://forums.phpfreaks.com/topic/188189-robotstxt/#findComment-993504 Share on other sites More sharing options...
liamloveslearning Posted January 12, 2010 Author Share Posted January 12, 2010 Thanks cags! Link to comment https://forums.phpfreaks.com/topic/188189-robotstxt/#findComment-993505 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.