ziegel Posted August 15, 2009 Share Posted August 15, 2009 What would be the fastest way to open many webpages and get the data from them? I am creating an item database for a game, and I am going to be looping through the items on their website to get the correct data to display in my item database. However, this game has around 15,000 items, and right now the speed is slow (I don't expect it to be fast, though). I've tried file_get_contents() and cURL, but each of them takes about 60-70 seconds to open EACH page. I'm thinking there has to be another way, since other sites with item databases for this game loop through this page for each item 2 or 3 times a day. Any ideas? Link to comment https://forums.phpfreaks.com/topic/170436-get-websites-page-contents-faster/ Share on other sites More sharing options...
Jessica Posted August 15, 2009 Share Posted August 15, 2009 What those sites probably did is get the data once, then store it in their own database to display on their site, that's why they're faster. Neopets FTW! Link to comment https://forums.phpfreaks.com/topic/170436-get-websites-page-contents-faster/#findComment-899048 Share on other sites More sharing options...
ziegel Posted August 15, 2009 Author Share Posted August 15, 2009 Well, this data updates 2 or so times a day, because it contains current going prices on the items, most prices change every 12 hours I think it is. Link to comment https://forums.phpfreaks.com/topic/170436-get-websites-page-contents-faster/#findComment-899059 Share on other sites More sharing options...
Jessica Posted August 15, 2009 Share Posted August 15, 2009 Then set up a cron job to check every 12 hours, and store the data. The point is to check their site as infrequently as possible. Link to comment https://forums.phpfreaks.com/topic/170436-get-websites-page-contents-faster/#findComment-899103 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.