sexbotz Posted October 19, 2013 Share Posted October 19, 2013 Looked on the web couldn't find it or I wasn't typing what should. I'm looking for information on how to write identifier code for your crawler bot so website owners can see that my bot crawled them in there awstats robot/spiders visitors. Currently I'm coming up as Unknown robot (identified by empty user agent string). Anyone know how to do this or where I could find some info on the subject? Quote Link to comment https://forums.phpfreaks.com/topic/283104-writing-an-identifier-for-my-crawler/ Share on other sites More sharing options...
.josh Posted October 19, 2013 Share Posted October 19, 2013 if you are using cURL then you can use curl_setopt and set CURLOPT_USERAGENT if you are using something more generic like file_get_contents then you will have to set it in your php.ini file Quote Link to comment https://forums.phpfreaks.com/topic/283104-writing-an-identifier-for-my-crawler/#findComment-1454565 Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.