master82 Posted August 17, 2006 Share Posted August 17, 2006 Hello all,Could somebody please explain how I would use the robots.txt to stop search engines indexing all of my site, except my index page.Could someone please give me an example of what i put in the robots.txt, and do I need to add anything to my HTML or PHP pages?Thanks in advance Quote Link to comment Share on other sites More sharing options...
Gruzin Posted August 17, 2006 Share Posted August 17, 2006 here:[color=red]User-agent: *Disallow: /apartments/redirect.phpDisallow: /nyc_apartments/new_york_city_rentals.htmDisallow: /includes/Disallow: /includes/appointment.phpDisallow: /includes/bookmark.phpAllow: /includes/img.phpAllow: /includes/gallery.php[/color] Quote Link to comment Share on other sites More sharing options...
master82 Posted August 17, 2006 Author Share Posted August 17, 2006 OK thanks, just 1 question:You haveDisallow: /includes/Disallow: /includes/appointments.phpAm I correct in thinking that you didnt need the second disallow above because you have already disallowed access to the folder? or has each page got to be disallowed individually? Quote Link to comment Share on other sites More sharing options...
Gruzin Posted August 17, 2006 Share Posted August 17, 2006 you are right, I wrote that for show. And you should add the robots code in to your HTML head:[color=red]<META NAME="robots" CONTENT="index, follow" />[/color] Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.