Jump to content

.htaccess and robots.txt


graham23s

Recommended Posts

Hi Guys,

 

Although i can code in PHP fairly well i'm using a 3rd party script for some of my sites for quickness.

 

In the .htaccess file is:

 

Options +FollowSymLinks
RewriteEngine on

#RewriteBase 

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

# General stuff
RewriteRule ^.htaccess$ - [F]
RewriteRule ^$ index.php
#RewriteRule ^.*/img/(.*\.(jpg|gif|png|swf))$ /img/$1
RewriteRule ^files/ - [L]
RewriteRule ^update\.php - [L]
RewriteRule ^updatefeed\.php - [L]
RewriteRule ^proxy\.php - [L]
RewriteRule ^install_img/ - [L]
RewriteRule ^themes/ - [L]
RewriteRule ^modules/ - [L]
RewriteRule ^js/ - [L]
RewriteRule ^js/tinymce/ - [L]
#RewriteRule ^js/scriptaculous/ - [L]
#RewriteRule ^.*/js/(.*\.js)$ /js/$1
#RewriteRule ^.*/css/(.*\.css)$ /css/$1
RewriteRule ^.*/ajax/(.*\.html)$ /ajax/$1
RewriteRule !(\.(js|css|gif|png|jpg|ico|swf))$ index.php

#ErrorDocument 404 /page/not-found/
<IfModule mod_php5.c>
php_flag magic_quotes_gpc Off
php_flag register_globals Off
php_flag short_open_tag on
php_value error_reporting 2047
php_value ignore_user_abort On
</IfModule>

 

The problem is, when i put a robots.txt file in and try to view it like: www.site.com/robots.txt it springs up a 404 (i assume it's because the .htaccess isn't allowing it)

 

any help would be appreciated guys!

 

cheers

 

Graham

Link to comment
Share on other sites

The only relevant rule in your .htaccess file seems to be this one...

 

RewriteRule !(\.(js|css|gif|png|jpg|ico|swf))$ index.php

 

So basically it is getting forwarded to the index.php, if you are receiving a 404 error, this is likely because the routing system in whatever CMS/software you are using doesn't recognise this as a valid route (understandable). There are several ways to go about rectifying this. Adding this line in before the aforementioned one is probably the simplest...

 

RewriteRule ^robots\.txt - [L] 

 

...or you could add .txt as a valid extension into that regex above.

 

Link to comment
Share on other sites

No problem, glad it's fixed. I should also point out that the two RewriteCond's at the start of the file are actually pretty pointless. RewriteCond's only apply to the RewriteRule directly following them. The objective of the Cond's is to make a Rule get skipped if a file doesn't exist, as such the two Conditions would be more properly located before the last RewriteRule you have. This would allow people to access files that actually exists on the server without getting redirected. Of course if all files that don't have a rule associated shouldn't be accessible then you could just scrap the Cond's altogether.

Link to comment
Share on other sites

This thread is more than a year old. Please don't revive it unless you have something important to add.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.