.htaccess and robots.txt

[graham23s]

Hi Guys,

 

Although i can code in PHP fairly well i'm using a 3rd party script for some of my sites for quickness.

 

In the .htaccess file is:

 

Options +FollowSymLinks
RewriteEngine on

#RewriteBase 

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

# General stuff
RewriteRule ^.htaccess$ - [F]
RewriteRule ^$ index.php
#RewriteRule ^.*/img/(.*\.(jpg|gif|png|swf))$ /img/$1
RewriteRule ^files/ - [L]
RewriteRule ^update\.php - [L]
RewriteRule ^updatefeed\.php - [L]
RewriteRule ^proxy\.php - [L]
RewriteRule ^install_img/ - [L]
RewriteRule ^themes/ - [L]
RewriteRule ^modules/ - [L]
RewriteRule ^js/ - [L]
RewriteRule ^js/tinymce/ - [L]
#RewriteRule ^js/scriptaculous/ - [L]
#RewriteRule ^.*/js/(.*\.js)$ /js/$1
#RewriteRule ^.*/css/(.*\.css)$ /css/$1
RewriteRule ^.*/ajax/(.*\.html)$ /ajax/$1
RewriteRule !(\.(js|css|gif|png|jpg|ico|swf))$ index.php

#ErrorDocument 404 /page/not-found/
<IfModule mod_php5.c>
php_flag magic_quotes_gpc Off
php_flag register_globals Off
php_flag short_open_tag on
php_value error_reporting 2047
php_value ignore_user_abort On
</IfModule>

 

The problem is, when i put a robots.txt file in and try to view it like: www.site.com/robots.txt it springs up a 404 (i assume it's because the .htaccess isn't allowing it)

 

any help would be appreciated guys!

 

cheers

 

Graham

[cags]

The only relevant rule in your .htaccess file seems to be this one...

 

RewriteRule !(\.(js|css|gif|png|jpg|ico|swf))$ index.php

 

So basically it is getting forwarded to the index.php, if you are receiving a 404 error, this is likely because the routing system in whatever CMS/software you are using doesn't recognise this as a valid route (understandable). There are several ways to go about rectifying this. Adding this line in before the aforementioned one is probably the simplest...

 

RewriteRule ^robots\.txt - [L] 

 

...or you could add .txt as a valid extension into that regex above.

 

[graham23s]

Thanks a ton Cags i added the extension in the regex as advised, everything is working great

 

thanks mate

 

Graham

[cags]

No problem, glad it's fixed. I should also point out that the two RewriteCond's at the start of the file are actually pretty pointless. RewriteCond's only apply to the RewriteRule directly following them. The objective of the Cond's is to make a Rule get skipped if a file doesn't exist, as such the two Conditions would be more properly located before the last RewriteRule you have. This would allow people to access files that actually exists on the server without getting redirected. Of course if all files that don't have a rule associated shouldn't be accessible then you could just scrap the Cond's altogether.