Azu Posted July 2, 2007 Share Posted July 2, 2007 Hello! Please tell me how I can tell the search engines to ignore a block of HTML. I have a login prompt on my website for anyone that isn't logged in, and I want to make the search engines ignore this, so that when you see my site in the search engine, there isn't a big fat "Please log in below" at the top of every result. Quote Link to comment Share on other sites More sharing options...
Daniel0 Posted July 2, 2007 Share Posted July 2, 2007 robots.txt Quote Link to comment Share on other sites More sharing options...
Azu Posted July 3, 2007 Author Share Posted July 3, 2007 How do you use robots.txt to tell search engines to ignore a certain section of on a page? Quote Link to comment Share on other sites More sharing options...
Daniel0 Posted July 3, 2007 Share Posted July 3, 2007 Make a file called robots.txt and place it in the document root. Put something like this in it: User-agent: * Disallow: /top_secret_stuff/ Disallow: /admin/ Disallow: /login.php Quote Link to comment Share on other sites More sharing options...
steviewdr Posted July 3, 2007 Share Posted July 3, 2007 I think Azu means getting bots to ignore parts of A PARTICULAR page. Azu - you will need to wrap the section in javascript, perhaps compress the section with javascript or something. Bots and robots etc. ignore javascript (well the googlebot anyways at least). -steve Quote Link to comment Share on other sites More sharing options...
Azu Posted July 4, 2007 Author Share Posted July 4, 2007 Thanks, the javascript solution works I think I am wandering though, is there a way I can just put a certain tag around the part I want the search engines to ignore, to make them ignore it? If I use the javascript solution, then it will only work in browsers that support javascript and have it enabled. This is bad for people that need screen readers and stuff. Quote Link to comment Share on other sites More sharing options...
Daniel0 Posted July 4, 2007 Share Posted July 4, 2007 You could use PHP to determine if the HTTP_USER_AGENT is a known bot (e.g. "googlebot"). If it is, then simply don't output it. Quote Link to comment Share on other sites More sharing options...
Azu Posted July 4, 2007 Author Share Posted July 4, 2007 Thanks, I was considering that, but then the spider won't follow the links. I just want it so that the block of text and stuff won't stuff up in the search result. I don't want to prevent it from following links in it though. I'm really stumped x_x Quote Link to comment Share on other sites More sharing options...
Daniel0 Posted July 4, 2007 Share Posted July 4, 2007 Consider this page: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>Title</title> </head> <body> <div>Everybody can see this.</div> <?php $bots = array('googlebot'); // i can't remember what the other ones are called if(!in_array($_SERVER['HTTP_USER_AGENT'], $bots)) { echo <<<EOF <div>You are not a known bot so you are allowed to view this as well.</div> EOF; } ?> <div>Everybody can see this as well.</div> </body> </html> In that way you can control what the bots should and shouldn't see (i.e. "blocking" text). Quote Link to comment Share on other sites More sharing options...
Azu Posted July 6, 2007 Author Share Posted July 6, 2007 Thank you Just 1 more little question.. is there a way to make it so that the search engine will still follow links in a block of text, but won't index the block of text? Quote Link to comment Share on other sites More sharing options...
Daniel0 Posted July 6, 2007 Share Posted July 6, 2007 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>Title</title> </head> <body> <div>Everybody can see this.</div> <?php $bots = array('googlebot'); // i can't remember what the other ones are called if(!in_array($_SERVER['HTTP_USER_AGENT'], $bots)) { // Don't display this to bots echo <<<EOF <div>You are not a known bot so you are allowed to view this as well.</div> EOF; } else { // display this to known bots only echo <<<EOF <div style='display:none;'><a href='http://example.com/page1.php>text here</a> <a href='http://example.com/page2.php'>other text here</a></div> EOF; } ?> <div>Everybody can see this as well.</div> </body> </html> Be careful with that though. You'll risk getting banned because it can look suspicious to a bot if you have a hidden div with a lot of links. I believe this is what is called "Cloaking". Quote Link to comment Share on other sites More sharing options...
Azu Posted July 6, 2007 Author Share Posted July 6, 2007 Thank you! This is what I am looking for. And it is just for making my navigation links not show up on all my search results, so I'm pretty sure I won't get in trouble for it. Quote Link to comment Share on other sites More sharing options...
steelmanronald06 Posted July 6, 2007 Share Posted July 6, 2007 Closed thread, i know, but this should help you with bot names: http://www.searchenginedictionary.com/spider-names.shtml Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.