Jump to content

Archived

This topic is now archived and is closed to further replies.

Hardbyte

GoogleBots, MSNBots, Bots in general.

Recommended Posts

Hi, and sorry for possibly posting in the wrong section.

Just wondering, do these spider bot things still trigger links within the page code even if I got a script to hide all links (or the whole page) if its equal to a bot? What Im trying to ask is, do they still spider through the code like a text file or do they work as if they were visiting like using a normal browser?

Any help would be appreciated.

Thanks.

Share this post


Link to post
Share on other sites
Just like a normal browser... They cant see what you write between the <?php and ?>, just the output.

Orio.

Share this post


Link to post
Share on other sites
Hi and thanks for your reply.

On another note: If I were to allow Mozilla as an agent, then the bots could still get thru. For example:

[code]MOZILLA/5.0 (COMPATIBLE; YAHOO! SLURP; HTTP://HELP.YAHOO.COM/HELP/US/YSEARCH/SLURP) has visited ....
MOZILLA/5.0 (COMPATIBLE; GOOGLEBOT/2.1; +HTTP://WWW.GOOGLE.COM/BOT.HTML) has visited ....[/code]

so just search for Yahoo & Google to block for example?

Thanks

Share this post


Link to post
Share on other sites
If your looking to block crawlers take a look at this: http://webtools.live2support.com/se_robots.php

You can place a text file in the directories and if there are rules, the bots are designed to follow the rules in the file before crawling your site.

Share this post


Link to post
Share on other sites
although i personally wouldn't rely 100% on that ^ those are rules that bots are "supposed" to follow, but that doesn't mean someone [i]has[/i] to program them to obey those rules.  I would implement that, but keep to your original plan of scripting to check for them.

Share this post


Link to post
Share on other sites
Thank you for the replies.

I agree with Crayon as Iv already got meta tags (no follow etc..) and robots.txt within the site which they are not following these "rules".

I dont mind them visiting the site, but they are triggering my "Report a comment" link and Im receiving [size=8pt]<backwards>[/size]tihs[size=8pt]</backwards>[/size] loads of emails.

My script seems to be working, its blocking the bots which I have in an array and those bots visiting which are not in the array - it emails me to inform me of the agent. (obviously allowing IE/Firefox and such common browsers.) Im just having to currently find out which are the bad bots  ???

Thanks again.

Share this post


Link to post
Share on other sites

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.