otuatail Posted September 12, 2008 Share Posted September 12, 2008 Hi > have been to the miscellaneous group without success. I have tried asking this before but got nowhere. I have a robots text file like. User-agent: Googlebot Disallow: /news/ Disallow: /cms/ User-agent: Slurp Disallow: /news/ Disallow: /cms/ This allows Google and Yahoo to crawl pages other than the ones listed above. My problem is if there is a new robot how do you get the User-agent string. For example I have been told to stop Yahoo use Slurp. how do you arive at that. The complete string is Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Am I to understand that I only need to use a part of this i.e. Slurp and that the Robots do a string test to see if Slurp is in there string. Or is there a seperate list of strings just for the robot.txt file. Desmond. P.S. I dont want to kill off all robots that would be dumb. I want to takle a few major ones and direct where they can go. Link to comment https://forums.phpfreaks.com/topic/123926-help-with-robottxt/ Share on other sites More sharing options...
Mr_J Posted September 12, 2008 Share Posted September 12, 2008 Sorry about this, but maybe it can help... http://www.webmasterworld.com/forum11/3023.htm http://www.webmasterworld.com/forum11/3086.htm Cheers Link to comment https://forums.phpfreaks.com/topic/123926-help-with-robottxt/#findComment-639740 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.