Looked on the web couldn't find it or I wasn't typing what should.
I'm looking for information on how to write identifier code for your crawler bot so website owners can see that my bot crawled them in there awstats robot/spiders visitors. Currently I'm coming up as Unknown robot (identified by empty user agent string).
Anyone know how to do this or where I could find some info on the subject?