Custom 404 page showing up in Google search

[andy1212]

Hey, I created a custom 404.php page and have the following code in the .htaccess to direct any non-existent pages to the custom 4040 page,

 

ErrorDocument 404 http://www.website.com/404.php

 

So it works great but when I search the website on google, 404.php is listed under the sitelinks. So I searched Google and came across their advice for what to do with a custom 404 page showing up in search and how to make it not show up.

 

"No matter how beautiful and useful your custom 404 page, you probably don't want it to appear in Google search results. In order to prevent 404 pages from being indexed by Google and other search engines, make sure that your webserver returns an actual 404 HTTP status code when a missing page is requested."

 

 

 

So then I searched Google for "actual 404 https status code" where there is lots of information about how it should work, but couldn't find any information about how to implement it. I have the 404.php page demoted under the sitelinks section of Google Webmaster Tools and there is a 0 soft 404 error under crawl errors. But do I need to put a line of code in the .htaccess file to show that an actual 404 error is there when on 404.php page.

 

Also, under Blocked URLs in webmaster tools there is this code,

 

 

User-agent: *

Crawl-delay: 10

 

on the robots.txt page and I'm not sure what that is.

 

Thanks for your time and help in advance!

[requinix]

Does your 404.php have a line

header("HTTP/1.1 404 Not Found");

in it? If so, what's the code for the file?

[andy1212]

No I don't have that line on the page, should I have something like that there? It's just a page exactly like the other pages of the website with a message saying woops... wrong page and then I have a header which redirects the page to index.php

[kicken]

Yes, you need to send back a proper 404 response status otherwise google and other bots will see it as a normal web page and index it as such.

[andy1212]

Ok I added this line to the top of the page,

 

header("HTTP/1.1 404 Not Found");

 

So would this solve the problem?

[requinix]

Maybe. Post your code.

[andy1212]

Hey I marked as solved too soon, I'm still getting 0 soft 404 error when Google crawls my website. I have the following code for the 404.php page,

<?
include 'log.php';
include 'info/404.info.php';
include 'header2.php';
header("HTTP/1.1 404 Not Found");
?>
<div id="legal">
<div id="middle2">
     <div id="leftcolumn">
         <div id="contactus">
                <div id="news" style="width: 550px; height: auto; float: left; text-align: left;">
                    <div style="margin-top: 30px; width: auto; height: auto; float: left;">
                     <span class="font3">Woops...</span><br /><br />
                        <span class="font15">
                        That page doesn't exist. Returning to the home page...
                        </span>
                    </div>
                </div>
            </div>
        </div>
        <div id="rightcolumn" style="text-align: left;">
         <div style="float: left; text-align: left; width: 400px; height: auto;">
         <span style="font-family: Arial, Helvetica, sans-serif; font-size: 20px; font-weight: bold; text-decoration: none; color: black;">Contact Us</span>
         <br /><br />
            <?
$id = '1';
$data = data7($id, 'street', 'city', 'phone', 'fax', 'email', 'url');
?>
            <span style="font-family: Arial, Helvetica, sans-serif; font-size: 12px; font-weight: normal; text-decoration: none; color: black;">
                <? echo  stripslashes($data['street']);?><br />
                <? echo stripslashes($data['city']); ?><br /><br />
                <b>Telephone:</b> <a href="#" class="font15"><? echo stripslashes($data['phone']); ?></a><br />
                <b>Fax:</b> <a href="#" class="font15"><? echo stripslashes($data['fax']); ?></a><br />
                <b>Email:</b> <? echo stripslashes($data['email']); ?> 
            </span>
            </div>
            <div style="float: left; text-align: left; width: 400px; height: auto; margin-top: 30px;">
            <span style="font-family: Arial, Helvetica, sans-serif; font-size: 20px; font-weight: bold; text-decoration: none; color: black;">Hours of Operation</span>
         <br /><br />
            <span style="font-family: Arial, Helvetica, sans-serif; font-size: 12px; font-weight: normal; text-decoration: none; color: black;">
            <?
$id = '1';
$data2 = data6($id, 'monday1', 'sat1', 'sun1', 'monday2', 'sat2', 'sun2');
?>
            <u>Spring & Summer Hours</u><br /><br />
            Monday - Friday: <? echo stripslashes($data2['monday1']); ?><br />
            Saturday: <? echo stripslashes($data2['sat1']); ?><br />
            Sunday: <? echo stripslashes($data2['sun1']); ?>
            <br /><br />
            <u>Fall & Winter Hours</u><br /><br />
            Monday - Friday: <? echo stripslashes($data2['monday2']); ?><br />
            Saturday: <? echo stripslashes($data2['sat2']); ?><br />
            Sunday: <? echo stripslashes($data2['sun2']); ?>
            </span>
            </div>
        </div>
    </div>
</div>
<?
include 'supplierlinks.php';
include 'footer.php';


header('refresh: 4; url=http://www.website.com');
exit();


?>

Thanks for your time and help

[requinix]

The header() has to happen before you've outputted anything. I'm guessing header2.php has some HTML? That's the problem.

 

And please use the long open tags <?php. Not everybody supports the short tags.

<?php
header("HTTP/1.1 404 Not Found");

include 'log.php';
include 'info/404.info.php';
include 'header2.php';
?>

[andy1212]

Ok thanks for the reply and advice!

[andy1212]

Also how do I check to see if the right header is being outputed?

[andy1212]

Ok now I'm getting 3 soft 404 error in Google web masters. I added the line, header("HTTP/1.1 404 Not Found"); to the very top of the page too and change all <? tags to <?php so not sure what's going wrong there now.

[kicken]

Also how do I check to see if the right header is being outputed?

Use the networking area of your favorite browser's dev tools to exam the request and response. It should show you the headers sent and headers received, including the status code.

 

Another option is to use something like Fiddler2 to exam the request/response.

[andy1212]

I checked safari and firefox for the networking area under the develop tools menu (safari) and web developer tools (firefox) and couldn't find it. What is the option I need to choose under Develop in Safari to view the header response. Thanks for your time!

[kicken]

I have no idea since I do not use safari, try Google. As for firefox you need to have Firebug installed and open, then you will see a tab labeled Net. You'll see the status and other headers as so:

[requinix]

Ok now I'm getting 3 soft 404 error in Google web masters. I added the line, header("HTTP/1.1 404 Not Found"); to the very top of the page too and change all tags to <?php so not sure what's going wrong there now.

Going wrong? A soft 404 is exactly what you want. It's right. Unless I'm missing something?

[andy1212]

This is what I have for the headers,

 

 

Cache-Control no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Connection Keep-Alive Content-Type text/html Date Wed, 07 Aug 2013 01:40:02 GMT Expires Thu, 19 Nov 1981 08:52:00 GMT Host-Header 192fc2e7e50945beb8231a492d6a8024 Keep-Alive timeout=5, max=100 Pragma no-cache Server Apache Transfer-Encoding chunked

[andy1212]

Oh ok, then ya I tried searching the website on google and 404.php didn't show up under site links! so I guess it's working now.

[andy1212]

Thanks kicken and requinix for all your help!

Edited August 7, 2013 by andy1212

[andy1212]

Ok **** it's still there under sitelinks. When I search on google while i'm logged in to my gmail account, 404.php doesn't show up in sitelinks under the snippet, but when I log out and do the same search 404.php shows up under site links. How do I get it to not show up in site links under the snippet?

[kicken]

Google takes time to update. Just because you fixed the problem doesn't mean it'll be fixed on google right away. It could take up to a few months for google to de-index your 404 page or any other urls you decide to exclude.

[requinix]

This is what I have for the headers,

Those don't matter. The first arrow on kicken's image points to the important part: the status code.

 

Ok **** it's still there under sitelinks. When I search on google while i'm logged in to my gmail account, 404.php doesn't show up in sitelinks under the snippet, but when I log out and do the same search 404.php shows up under site links. How do I get it to not show up in site links under the snippet?

What does GMail have to do with any of this? Sitelinks where? Regardless of that, keep in mind that Google doesn't remove pages from their index immediately.

[andy1212]

oh ok