Jump to content

How can i obtain better performance with this?


bores_escalovsk

Recommended Posts

Here is the deal, its a simple script.

 

1-read a site link from a sql table**(sql1)

2-mark the sql1 line as read

3-goes to the site and capture several pieces of data(i used curl)

4-modify the read data

5-writes the data in another sql(sql2)

 

So i did this with a few links, but i have to do this with 5~10 millions of links, what would be the better way to get performance and speed?

Link to comment
Share on other sites

You'll want to add some concurrency to the process. The easiest place would be in the downloading stage using a library such as Rolling-CURL to download several URLs at once.

 

If you're feeling adventurous then you could also look into pcntl_fork to create multiple processes which will process urls/results concurrently as well.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.