otuatail Posted March 3, 2007 Share Posted March 3, 2007 hi I am trying to controll robots. I don't want to disalow them in the robot.txt file, but I do wan't to know if the visit. This helps me to stop things like increase hit count and serve up a page that gets them more excited. I have a MySQL database that logs all entries to each page as well as a first visit. What I want is a fairly up to date list of robots including there agent string ("HTTP_USER_AGENT") and possibly the name that would be included in a robot.txt file. Any Help ??? Desmond. Link to comment https://forums.phpfreaks.com/topic/40993-web-robots/ Share on other sites More sharing options...
TRI0N Posted March 3, 2007 Share Posted March 3, 2007 Robots Database. http://www.robotstxt.org/wc/active/html/index.html Link to comment https://forums.phpfreaks.com/topic/40993-web-robots/#findComment-198604 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.