Hardbyte Posted July 6, 2006 Share Posted July 6, 2006 Hi, and sorry for possibly posting in the wrong section.Just wondering, do these spider bot things still trigger links within the page code even if I got a script to hide all links (or the whole page) if its equal to a bot? What Im trying to ask is, do they still spider through the code like a text file or do they work as if they were visiting like using a normal browser?Any help would be appreciated.Thanks. Quote Link to comment https://forums.phpfreaks.com/topic/13826-googlebots-msnbots-bots-in-general/ Share on other sites More sharing options...
Orio Posted July 6, 2006 Share Posted July 6, 2006 Just like a normal browser... They cant see what you write between the <?php and ?>, just the output.Orio. Quote Link to comment https://forums.phpfreaks.com/topic/13826-googlebots-msnbots-bots-in-general/#findComment-53750 Share on other sites More sharing options...
Hardbyte Posted July 6, 2006 Author Share Posted July 6, 2006 Hi and thanks for your reply.On another note: If I were to allow Mozilla as an agent, then the bots could still get thru. For example:[code]MOZILLA/5.0 (COMPATIBLE; YAHOO! SLURP; HTTP://HELP.YAHOO.COM/HELP/US/YSEARCH/SLURP) has visited ....MOZILLA/5.0 (COMPATIBLE; GOOGLEBOT/2.1; +HTTP://WWW.GOOGLE.COM/BOT.HTML) has visited ....[/code]so just search for Yahoo & Google to block for example?Thanks Quote Link to comment https://forums.phpfreaks.com/topic/13826-googlebots-msnbots-bots-in-general/#findComment-53775 Share on other sites More sharing options...
SharkBait Posted July 6, 2006 Share Posted July 6, 2006 If your looking to block crawlers take a look at this: http://webtools.live2support.com/se_robots.phpYou can place a text file in the directories and if there are rules, the bots are designed to follow the rules in the file before crawling your site. Quote Link to comment https://forums.phpfreaks.com/topic/13826-googlebots-msnbots-bots-in-general/#findComment-53898 Share on other sites More sharing options...
.josh Posted July 6, 2006 Share Posted July 6, 2006 although i personally wouldn't rely 100% on that ^ those are rules that bots are "supposed" to follow, but that doesn't mean someone [i]has[/i] to program them to obey those rules. I would implement that, but keep to your original plan of scripting to check for them. Quote Link to comment https://forums.phpfreaks.com/topic/13826-googlebots-msnbots-bots-in-general/#findComment-53902 Share on other sites More sharing options...
Hardbyte Posted July 6, 2006 Author Share Posted July 6, 2006 Thank you for the replies.I agree with Crayon as Iv already got meta tags (no follow etc..) and robots.txt within the site which they are not following these "rules".I dont mind them visiting the site, but they are triggering my "Report a comment" link and Im receiving [size=8pt]<backwards>[/size]tihs[size=8pt]</backwards>[/size] loads of emails.My script seems to be working, its blocking the bots which I have in an array and those bots visiting which are not in the array - it emails me to inform me of the agent. (obviously allowing IE/Firefox and such common browsers.) Im just having to currently find out which are the bad bots ???Thanks again. Quote Link to comment https://forums.phpfreaks.com/topic/13826-googlebots-msnbots-bots-in-general/#findComment-53911 Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.