marcin_koss Posted February 17, 2010 Share Posted February 17, 2010 I have a website advertising beer products that has age verification page. After visitor confirms that he or she is 21+ years old, $_SESSION variable is set and the visitor is able to view the content of the website. Recently I found that the content from the pages behind age verification page are not scanned by search engine crawlers. It looks like crawlers can't get through age verification page. Does this has something to do with using PHP's $_SESSION variables - should I use different approach for age verification? I'm so puzzled. Thanks. Quote Link to comment https://forums.phpfreaks.com/topic/192390-search-engine-crawlers-and-protected-pages/ Share on other sites More sharing options...
premiso Posted February 17, 2010 Share Posted February 17, 2010 It is, as most bots will not care to set cookies and without a cookie they cannot gain access, that and they probably do not wish to confirm their age. You can check the useragent and if the useragent is a known bot (you can find bot agent strings online) allow them access without verification. Quote Link to comment https://forums.phpfreaks.com/topic/192390-search-engine-crawlers-and-protected-pages/#findComment-1013749 Share on other sites More sharing options...
marcin_koss Posted February 17, 2010 Author Share Posted February 17, 2010 Thanks for response. These bots have the most amount of hits to my page. Should I look for scripts for each of them? Yandex bot, MSNBot, Googlebot, Yahoo Slurp How about going with Javascript based verification? Quote Link to comment https://forums.phpfreaks.com/topic/192390-search-engine-crawlers-and-protected-pages/#findComment-1013801 Share on other sites More sharing options...
SchweppesAle Posted February 17, 2010 Share Posted February 17, 2010 It is, as most bots will not care to set cookies and without a cookie they cannot gain access, that and they probably do not wish to confirm their age. You can check the useragent and if the useragent is a known bot (you can find bot agent strings online) allow them access without verification. pretty sure $_SESSION is server side. Quote Link to comment https://forums.phpfreaks.com/topic/192390-search-engine-crawlers-and-protected-pages/#findComment-1013965 Share on other sites More sharing options...
premiso Posted February 17, 2010 Share Posted February 17, 2010 pretty sure $_SESSION is server side. The data is stored on the server, however, the session ID has to be passed to the server to authenticate a user's access to that data. This is generally set in a cookie and or if it is setup to do so can be appended with GET data incase cookies have been disabled on the user's end. Quote Link to comment https://forums.phpfreaks.com/topic/192390-search-engine-crawlers-and-protected-pages/#findComment-1013985 Share on other sites More sharing options...
shlumph Posted February 17, 2010 Share Posted February 17, 2010 pretty sure $_SESSION is server side. The data values are stored on the server, however, the client is given a unique id in the form of a cookie. That's why when you "clean private data" on your web browser, the server can't find your $_SESSION. Web crawlers simply crawl your public pages. They cannot submit your form; knowing to be the age of 21 or older, or anything fancy like that. I'm not sure what the best solution for you would be. Quote Link to comment https://forums.phpfreaks.com/topic/192390-search-engine-crawlers-and-protected-pages/#findComment-1013989 Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.