ziegel Posted August 15, 2009 Share Posted August 15, 2009 What would be the fastest way to open many webpages and get the data from them? I am creating an item database for a game, and I am going to be looping through the items on their website to get the correct data to display in my item database. However, this game has around 15,000 items, and right now the speed is slow (I don't expect it to be fast, though). I've tried file_get_contents() and cURL, but each of them takes about 60-70 seconds to open EACH page. I'm thinking there has to be another way, since other sites with item databases for this game loop through this page for each item 2 or 3 times a day. Any ideas? Quote Link to comment Share on other sites More sharing options...
Jessica Posted August 15, 2009 Share Posted August 15, 2009 What those sites probably did is get the data once, then store it in their own database to display on their site, that's why they're faster. Neopets FTW! Quote Link to comment Share on other sites More sharing options...
ziegel Posted August 15, 2009 Author Share Posted August 15, 2009 Well, this data updates 2 or so times a day, because it contains current going prices on the items, most prices change every 12 hours I think it is. Quote Link to comment Share on other sites More sharing options...
Jessica Posted August 15, 2009 Share Posted August 15, 2009 Then set up a cron job to check every 12 hours, and store the data. The point is to check their site as infrequently as possible. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.