"slowly"Walking trough a list with links using CURL while giving visual feedback

[Axeia]

Okay, guess it's a bit too long to squeeze all in the subject line

 

I got a list of links in my database with a counter field for the amount of times it's been attempted to download. Defaulting to 0.

Now the idea is to slowly walk trough this list (to avoid getting banned from sites by flooding them with requests) while giving back a visual indication of what's going on. (without relying on ajax even)

 

So my idea was to have 2 iframes

1) A self refreshing page(via the meta tag and a ~30 sec delay or so) with the visual feedback a, list with the last 10 added values. And all results with count = 0

2) Hidden iframe initiating the process of walking trough the list like this

* SELECT blabla from blabla WHERE count = 0 AND limit = 1;

* Get the blabla and feed it to curl, and if the retrieved content was page could not be found or something update the count +1. If it was succesfull move it to another table.

* Sleep for a second and check in the database if there anything with count = 0 left, if so let it curl itself.

 

Good way to go about it, or is there a better way?

 

[Daniel0]

I don't completely understand how you want it to work, but can't you use sleep to make sure you wait at least a fixed amount of seconds between each request?

[Axeia]

I don't completely understand how you want it to work, but can't you use sleep to make sure you wait at least a fixed amount of seconds between each request?

Sleep for a second and check in the database if there anything with count = 0 left, if so let it curl itself."

So yeah, that was the plan, can't do that on the page itself as it would result in a slow loading page refreshing all the time.

 

I forgot to mention btw that an hourly cron job is adding the links to the database and I planned on letting that take care of the links where the count is greater than 1. Up till a yet to determine number where the link will switch its status to "impossible" or something as it might just be a lost cause if isn't up in say ~2 days.