Jump to content


Photo

robots.txt


  • Please log in to reply
3 replies to this topic

#1 master82

master82
  • Members
  • PipPipPip
  • Advanced Member
  • 182 posts

Posted 17 August 2006 - 01:14 PM

Hello all,

Could somebody please explain how I would use the robots.txt to stop search engines indexing all of my site, except my index page.

Could someone please give me an example of what i put in the robots.txt, and do I need to add anything to my HTML or PHP pages?

Thanks in advance

#2 Gruzin

Gruzin
  • Members
  • PipPipPip
  • Advanced Member
  • 448 posts
  • LocationGeorgia

Posted 17 August 2006 - 01:15 PM

here:

User-agent: *
Disallow: /apartments/redirect.php
Disallow: /nyc_apartments/new_york_city_rentals.htm
Disallow: /includes/
Disallow: /includes/appointment.php
Disallow: /includes/bookmark.php
Allow: /includes/img.php
Allow: /includes/gallery.php

I don't need your script, I'll try to write it myself

#3 master82

master82
  • Members
  • PipPipPip
  • Advanced Member
  • 182 posts

Posted 17 August 2006 - 01:21 PM

OK thanks, just 1 question:

You have

Disallow: /includes/
Disallow: /includes/appointments.php

Am I correct in thinking that you didnt need the second disallow above because you have already disallowed access to the folder? or has each page got to be disallowed individually?

#4 Gruzin

Gruzin
  • Members
  • PipPipPip
  • Advanced Member
  • 448 posts
  • LocationGeorgia

Posted 17 August 2006 - 01:24 PM

you are right, I wrote that for show. And you should add the robots code in to your HTML head:


<META NAME="robots" CONTENT="index, follow" />
I don't need your script, I'll try to write it myself




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users