Allowing Search Engine Spiders In

[Smudged123]

I'm currently running an adult-themed site and started noticing problems when I try to create a sitemap for my site. It only seems to index the first page, which is a PHP age verification page (validuser.php) and is initiated from header.php when the page loads with:

// Splash Page request...
if(!(isset($_COOKIE["valid_user_age"]) && $_COOKIE["valid_user_age"] == "y")){
redirect(HOST . '/validuser.php');
} // if

 

I was wondering if I could use some sort of function that would verify the HTTP_USER_AGENT when the page is loaded and if it were anything besides a normal user then to automatically set the cookie for age verification?

Or are there any similar fixes to this problem?

Thanks!!

[Snart]

http://iarematt.com/how-to-detect-a-search-engine-spidercrawler-with-php/

[Smudged123]

Alright so I came up with...

if ( ! function_exists('check_if_spider'))  
{  
    function check_if_spider()  
    {  
        // Add as many spiders you want in this array  
        $spiders    = array(  
                        'Googlebot', 'Yammybot', 'Openbot', 'Yahoo', 'Slurp', 'msnbot',  
                        'ia_archiver', 'Lycos', 'Scooter', 'AltaVista', 'Teoma', 'Gigabot',  
                        'Googlebot-Mobile'  
                    );  
  
        // Loop through each spider and check if it appears in  
        // the User Agent  
        foreach ($spiders as $spider)  
        {  
            if (eregi($spider, $_SERVER['HTTP_USER_AGENT']))  
            {  
                	setcookie("valid_user_age", "y", time()+(3600*24), "/" );
		redirect(HOST . '/');
		return TRUE;  
            }  
        }  
        return FALSE;  
    }  
}

 

Basically just took the code from the site you cited for me and added my setcookie function if it detects it's a search engine spider.

 

Is it even possible to set a cookie for a search engine spider like this?

 

[Snart]

Cookies are browser-initiated textfiles stored on the local system.  I doubt that spiders meet the requirements to accept and keep cookies.

[Smudged123]

So given having the script that can detect search engine spiders, and my validuser.php code snippet as shown below:

	if($cal_timestamp >= (31556926*18)){
	setcookie("valid_user_age", "y", time()+(3600*24), "/" );
	redirect(HOST . '/');
}else{
	$msg = '<font color="red">You are Under 18 ,Please exit</font>';
} // if

 

Would there be a way for me to set it so if a search engine spider is detected then the cookie is not required to view the rest of the site?

Thanks