Jump to content

Server Timeout Scraping Multiple Pages


shaunie

Recommended Posts

Hi,

 

I have some code which scrapes data from a page. However there are around 1200 product pages on the site I need to scrape, when I attempt to loop through all the pages I get a server timeout. I can only get to around 40 without timeout. Has anyone else had this problem?

Link to comment
https://forums.phpfreaks.com/topic/228000-server-timeout-scraping-multiple-pages/
Share on other sites

Even more than that... based on network speeds at the time, the time to complete could vary greatly. The site does not provide any other way to get the product data? You think maybe they don't want people to scrape their pages?

 

:)

 

Just a thought.

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.