bores_escalovsk Posted May 9, 2014 Share Posted May 9, 2014 (edited) Here is the deal, its a simple script. 1-read a site link from a sql table**(sql1) 2-mark the sql1 line as read 3-goes to the site and capture several pieces of data(i used curl) 4-modify the read data 5-writes the data in another sql(sql2) So i did this with a few links, but i have to do this with 5~10 millions of links, what would be the better way to get performance and speed? Edited May 9, 2014 by bores_escalovsk Quote Link to comment https://forums.phpfreaks.com/topic/288372-how-can-i-obtain-better-performance-with-this/ Share on other sites More sharing options...
Jacques1 Posted May 9, 2014 Share Posted May 9, 2014 What on earth is an “sql file”? Quote Link to comment https://forums.phpfreaks.com/topic/288372-how-can-i-obtain-better-performance-with-this/#findComment-1478891 Share on other sites More sharing options...
ginerjm Posted May 9, 2014 Share Posted May 9, 2014 Other than moving step 2 to the end so that you are assured that the process is complete, what's wrong with this approach? (assuming that an sql file is a table) Quote Link to comment https://forums.phpfreaks.com/topic/288372-how-can-i-obtain-better-performance-with-this/#findComment-1478892 Share on other sites More sharing options...
bores_escalovsk Posted May 9, 2014 Author Share Posted May 9, 2014 What on earth is an “sql file”? it reads from sql database ** Quote Link to comment https://forums.phpfreaks.com/topic/288372-how-can-i-obtain-better-performance-with-this/#findComment-1478893 Share on other sites More sharing options...
bores_escalovsk Posted May 9, 2014 Author Share Posted May 9, 2014 Other than moving step 2 to the end so that you are assured that the process is complete, what's wrong with this approach? (assuming that an sql file is a table) im not sure about how much time, and my script is not multi tasking.. Quote Link to comment https://forums.phpfreaks.com/topic/288372-how-can-i-obtain-better-performance-with-this/#findComment-1478894 Share on other sites More sharing options...
ginerjm Posted May 9, 2014 Share Posted May 9, 2014 I might want to read batches of records and then process and write them back, either one at a time or in batches again. Quote Link to comment https://forums.phpfreaks.com/topic/288372-how-can-i-obtain-better-performance-with-this/#findComment-1478895 Share on other sites More sharing options...
Solution kicken Posted May 9, 2014 Solution Share Posted May 9, 2014 You'll want to add some concurrency to the process. The easiest place would be in the downloading stage using a library such as Rolling-CURL to download several URLs at once. If you're feeling adventurous then you could also look into pcntl_fork to create multiple processes which will process urls/results concurrently as well. Quote Link to comment https://forums.phpfreaks.com/topic/288372-how-can-i-obtain-better-performance-with-this/#findComment-1478915 Share on other sites More sharing options...
bores_escalovsk Posted May 9, 2014 Author Share Posted May 9, 2014 really good ideas, my php is not good enough for none of them yet, i will have to learn several things to do this right. time to start reading Quote Link to comment https://forums.phpfreaks.com/topic/288372-how-can-i-obtain-better-performance-with-this/#findComment-1478916 Share on other sites More sharing options...
bores_escalovsk Posted May 9, 2014 Author Share Posted May 9, 2014 this is what i call a forum Quote Link to comment https://forums.phpfreaks.com/topic/288372-how-can-i-obtain-better-performance-with-this/#findComment-1478919 Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.