Azu Posted July 2, 2007 Share Posted July 2, 2007 Hello! Please tell me how I can tell the search engines to ignore a block of HTML. I have a login prompt on my website for anyone that isn't logged in, and I want to make the search engines ignore this, so that when you see my site in the search engine, there isn't a big fat "Please log in below" at the top of every result. Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/ Share on other sites More sharing options...
Daniel0 Posted July 2, 2007 Share Posted July 2, 2007 robots.txt Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-287697 Share on other sites More sharing options...
Azu Posted July 3, 2007 Author Share Posted July 3, 2007 How do you use robots.txt to tell search engines to ignore a certain section of on a page? Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-288448 Share on other sites More sharing options...
Daniel0 Posted July 3, 2007 Share Posted July 3, 2007 Make a file called robots.txt and place it in the document root. Put something like this in it: User-agent: * Disallow: /top_secret_stuff/ Disallow: /admin/ Disallow: /login.php Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-288577 Share on other sites More sharing options...
steviewdr Posted July 3, 2007 Share Posted July 3, 2007 I think Azu means getting bots to ignore parts of A PARTICULAR page. Azu - you will need to wrap the section in javascript, perhaps compress the section with javascript or something. Bots and robots etc. ignore javascript (well the googlebot anyways at least). -steve Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-288615 Share on other sites More sharing options...
Azu Posted July 4, 2007 Author Share Posted July 4, 2007 Thanks, the javascript solution works I think I am wandering though, is there a way I can just put a certain tag around the part I want the search engines to ignore, to make them ignore it? If I use the javascript solution, then it will only work in browsers that support javascript and have it enabled. This is bad for people that need screen readers and stuff. Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-289433 Share on other sites More sharing options...
Daniel0 Posted July 4, 2007 Share Posted July 4, 2007 You could use PHP to determine if the HTTP_USER_AGENT is a known bot (e.g. "googlebot"). If it is, then simply don't output it. Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-289478 Share on other sites More sharing options...
Azu Posted July 4, 2007 Author Share Posted July 4, 2007 Thanks, I was considering that, but then the spider won't follow the links. I just want it so that the block of text and stuff won't stuff up in the search result. I don't want to prevent it from following links in it though. I'm really stumped x_x Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-289638 Share on other sites More sharing options...
Daniel0 Posted July 4, 2007 Share Posted July 4, 2007 Consider this page: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>Title</title> </head> <body> <div>Everybody can see this.</div> <?php $bots = array('googlebot'); // i can't remember what the other ones are called if(!in_array($_SERVER['HTTP_USER_AGENT'], $bots)) { echo <<<EOF <div>You are not a known bot so you are allowed to view this as well.</div> EOF; } ?> <div>Everybody can see this as well.</div> </body> </html> In that way you can control what the bots should and shouldn't see (i.e. "blocking" text). Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-289788 Share on other sites More sharing options...
Azu Posted July 6, 2007 Author Share Posted July 6, 2007 Thank you Just 1 more little question.. is there a way to make it so that the search engine will still follow links in a block of text, but won't index the block of text? Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-290983 Share on other sites More sharing options...
Daniel0 Posted July 6, 2007 Share Posted July 6, 2007 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>Title</title> </head> <body> <div>Everybody can see this.</div> <?php $bots = array('googlebot'); // i can't remember what the other ones are called if(!in_array($_SERVER['HTTP_USER_AGENT'], $bots)) { // Don't display this to bots echo <<<EOF <div>You are not a known bot so you are allowed to view this as well.</div> EOF; } else { // display this to known bots only echo <<<EOF <div style='display:none;'><a href='http://example.com/page1.php>text here</a> <a href='http://example.com/page2.php'>other text here</a></div> EOF; } ?> <div>Everybody can see this as well.</div> </body> </html> Be careful with that though. You'll risk getting banned because it can look suspicious to a bot if you have a hidden div with a lot of links. I believe this is what is called "Cloaking". Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-291012 Share on other sites More sharing options...
Azu Posted July 6, 2007 Author Share Posted July 6, 2007 Thank you! This is what I am looking for. And it is just for making my navigation links not show up on all my search results, so I'm pretty sure I won't get in trouble for it. Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-291331 Share on other sites More sharing options...
steelmanronald06 Posted July 6, 2007 Share Posted July 6, 2007 Closed thread, i know, but this should help you with bot names: http://www.searchenginedictionary.com/spider-names.shtml Link to comment https://forums.phpfreaks.com/topic/58034-solved-how-to-tell-search-engines-to-ignore-part-of-html/#findComment-291417 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.