onlyican Posted February 5, 2010 Share Posted February 5, 2010 Hi. NOTE: Please read this fully before replying. Problem: Database containing over 5000 URLs which has been created over several years. Over the years Website pages change (301, 302, 303 headers) Websites get put up for sale Websites go Offline Solution: Build a Spider to crawl the websites, checking the current status Methodology Websites returning a 200 OK header, are marked as OK Websites Returning NO Header marked as offline (Website down, could be temp, try again later My Problem Working out websites with a 301 as the page has moved Working out websites which have gone into parking or for sale. Does anyone hold an algorithm to help calculate if a website has been put up for sale or is now in parking. Or any ideas on what to look for. Link to comment https://forums.phpfreaks.com/topic/191039-website-status-checks/ Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.