hackalive Posted June 8, 2010 Share Posted June 8, 2010 Hello I am looking to build a web crawler such as MLBot. It must recognise robots.txt and ROBOTS meta tag, but in saying that when a site such as Wordpress shows visitor stats it lists the crawler (eg MLBot http://www.metadatalabs.com/mlbot). So how can I build a crawler that will list as HackAliveBot or HackAliveCrawler and will recognise robots.txt and ROBOTS meta tag. Thanks so much in advance Link to comment https://forums.phpfreaks.com/topic/204164-crawler/ Share on other sites More sharing options...
trq Posted June 8, 2010 Share Posted June 8, 2010 This board is for help with specific programming questions, your post is fare too vague and too widely scoped to possibly be answered within a simple forum post. I suggest you start posting these questions elsewhere. Link to comment https://forums.phpfreaks.com/topic/204164-crawler/#findComment-1069306 Share on other sites More sharing options...
Recommended Posts