Jump to content

GoogleBots, MSNBots, Bots in general.


Hardbyte

Recommended Posts

Hi, and sorry for possibly posting in the wrong section.

Just wondering, do these spider bot things still trigger links within the page code even if I got a script to hide all links (or the whole page) if its equal to a bot? What Im trying to ask is, do they still spider through the code like a text file or do they work as if they were visiting like using a normal browser?

Any help would be appreciated.

Thanks.
Link to comment
Share on other sites

Hi and thanks for your reply.

On another note: If I were to allow Mozilla as an agent, then the bots could still get thru. For example:

[code]MOZILLA/5.0 (COMPATIBLE; YAHOO! SLURP; HTTP://HELP.YAHOO.COM/HELP/US/YSEARCH/SLURP) has visited ....
MOZILLA/5.0 (COMPATIBLE; GOOGLEBOT/2.1; +HTTP://WWW.GOOGLE.COM/BOT.HTML) has visited ....[/code]

so just search for Yahoo & Google to block for example?

Thanks
Link to comment
Share on other sites

If your looking to block crawlers take a look at this: http://webtools.live2support.com/se_robots.php

You can place a text file in the directories and if there are rules, the bots are designed to follow the rules in the file before crawling your site.
Link to comment
Share on other sites

Thank you for the replies.

I agree with Crayon as Iv already got meta tags (no follow etc..) and robots.txt within the site which they are not following these "rules".

I dont mind them visiting the site, but they are triggering my "Report a comment" link and Im receiving [size=8pt]<backwards>[/size]tihs[size=8pt]</backwards>[/size] loads of emails.

My script seems to be working, its blocking the bots which I have in an array and those bots visiting which are not in the array - it emails me to inform me of the agent. (obviously allowing IE/Firefox and such common browsers.) Im just having to currently find out which are the bad bots  ???

Thanks again.
Link to comment
Share on other sites

This thread is more than a year old. Please don't revive it unless you have something important to add.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.