hamza Posted August 24, 2012 Share Posted August 24, 2012 please see me robot file # global Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/cache/ Disallow: /wp-content/themes/ Disallow: /trackback/ Disallow: /feed/ Disallow: /comments/ Disallow: /category/*/* Disallow: */trackback/ Disallow: */feed/ Disallow: */comments/ Disallow: /*? Allow: /wp-content/uploads/ #And here it is my subdir # Disallow Other subdirs Disallow: /test/ Disallow: /livelimmitless/ Disallow: /demos/ The Issue robots not crawlering on my site. and not indexing it . please see the file and help me with this what excactly the issue is. thanks Quote Link to comment https://forums.phpfreaks.com/topic/267536-robottxt/ Share on other sites More sharing options...
ManiacDan Posted August 24, 2012 Share Posted August 24, 2012 This disallows almost every page of a WP install. Specifically this line: Disallow: /*? Unless you have mod_rewrite enabled, this blocks everything. What do the URLs to your site look like? Quote Link to comment https://forums.phpfreaks.com/topic/267536-robottxt/#findComment-1372212 Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.