Jump to content

Archived

This topic is now archived and is closed to further replies.

master82

robots.txt

Recommended Posts

Hello all,

Could somebody please explain how I would use the robots.txt to stop search engines indexing all of my site, except my index page.

Could someone please give me an example of what i put in the robots.txt, and do I need to add anything to my HTML or PHP pages?

Thanks in advance

Share this post


Link to post
Share on other sites
here:

[color=red]User-agent: *
Disallow: /apartments/redirect.php
Disallow: /nyc_apartments/new_york_city_rentals.htm
Disallow: /includes/
Disallow: /includes/appointment.php
Disallow: /includes/bookmark.php
Allow: /includes/img.php
Allow: /includes/gallery.php[/color]

Share this post


Link to post
Share on other sites
OK thanks, just 1 question:

You have

Disallow: /includes/
Disallow: /includes/appointments.php

Am I correct in thinking that you didnt need the second disallow above because you have already disallowed access to the folder? or has each page got to be disallowed individually?

Share this post


Link to post
Share on other sites
you are right, I wrote that for show. And you should add the robots code in to your HTML head:


[color=red]<META NAME="robots" CONTENT="index, follow" />[/color]

Share this post


Link to post
Share on other sites

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.