otuatail Posted September 12, 2008 Share Posted September 12, 2008 Hi > have been to the miscellaneous group without success. I have tried asking this before but got nowhere. I have a robots text file like. User-agent: Googlebot Disallow: /news/ Disallow: /cms/ User-agent: Slurp Disallow: /news/ Disallow: /cms/ This allows Google and Yahoo to crawl pages other than the ones listed above. My problem is if there is a new robot how do you get the User-agent string. For example I have been told to stop Yahoo use Slurp. how do you arive at that. The complete string is Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Am I to understand that I only need to use a part of this i.e. Slurp and that the Robots do a string test to see if Slurp is in there string. Or is there a seperate list of strings just for the robot.txt file. Desmond. P.S. I dont want to kill off all robots that would be dumb. I want to takle a few major ones and direct where they can go. Quote Link to comment Share on other sites More sharing options...
Mr_J Posted September 12, 2008 Share Posted September 12, 2008 Sorry about this, but maybe it can help... http://www.webmasterworld.com/forum11/3023.htm http://www.webmasterworld.com/forum11/3086.htm Cheers Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.