master82 Posted August 17, 2006 Share Posted August 17, 2006 Hello all,Could somebody please explain how I would use the robots.txt to stop search engines indexing all of my site, except my index page.Could someone please give me an example of what i put in the robots.txt, and do I need to add anything to my HTML or PHP pages?Thanks in advance Link to comment https://forums.phpfreaks.com/topic/17835-robotstxt/ Share on other sites More sharing options...
Gruzin Posted August 17, 2006 Share Posted August 17, 2006 here:[color=red]User-agent: *Disallow: /apartments/redirect.phpDisallow: /nyc_apartments/new_york_city_rentals.htmDisallow: /includes/Disallow: /includes/appointment.phpDisallow: /includes/bookmark.phpAllow: /includes/img.phpAllow: /includes/gallery.php[/color] Link to comment https://forums.phpfreaks.com/topic/17835-robotstxt/#findComment-76154 Share on other sites More sharing options...
master82 Posted August 17, 2006 Author Share Posted August 17, 2006 OK thanks, just 1 question:You haveDisallow: /includes/Disallow: /includes/appointments.phpAm I correct in thinking that you didnt need the second disallow above because you have already disallowed access to the folder? or has each page got to be disallowed individually? Link to comment https://forums.phpfreaks.com/topic/17835-robotstxt/#findComment-76155 Share on other sites More sharing options...
Gruzin Posted August 17, 2006 Share Posted August 17, 2006 you are right, I wrote that for show. And you should add the robots code in to your HTML head:[color=red]<META NAME="robots" CONTENT="index, follow" />[/color] Link to comment https://forums.phpfreaks.com/topic/17835-robotstxt/#findComment-76157 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.