shaunie Posted February 17, 2011 Share Posted February 17, 2011 Hi, I have some code which scrapes data from a page. However there are around 1200 product pages on the site I need to scrape, when I attempt to loop through all the pages I get a server timeout. I can only get to around 40 without timeout. Has anyone else had this problem? Link to comment https://forums.phpfreaks.com/topic/228000-server-timeout-scraping-multiple-pages/ Share on other sites More sharing options...
BlueSkyIS Posted February 17, 2011 Share Posted February 17, 2011 I suspect it is php configuration of max_execution_time. you might get around this using set_time_limit() at the top of your code: http://php.net/manual/en/function.set-time-limit.php Link to comment https://forums.phpfreaks.com/topic/228000-server-timeout-scraping-multiple-pages/#findComment-1175716 Share on other sites More sharing options...
shaunie Posted February 17, 2011 Author Share Posted February 17, 2011 Hi, Thanks for your reply, the problem with this is that I won't know how long i need to increase the time by as the product list could grow. Can PHP handle multi threading? Link to comment https://forums.phpfreaks.com/topic/228000-server-timeout-scraping-multiple-pages/#findComment-1175794 Share on other sites More sharing options...
MatthewJ Posted February 17, 2011 Share Posted February 17, 2011 Even more than that... based on network speeds at the time, the time to complete could vary greatly. The site does not provide any other way to get the product data? You think maybe they don't want people to scrape their pages? Just a thought. Link to comment https://forums.phpfreaks.com/topic/228000-server-timeout-scraping-multiple-pages/#findComment-1175798 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.