Jump to content

Archived

This topic is now archived and is closed to further replies.

Axeia

"slowly"Walking trough a list with links using CURL while giving visual feedback

Recommended Posts

Okay, guess it's a bit too long to squeeze all in the subject line :)

 

I got a list of links in my database with a counter field for the amount of times it's been attempted to download. Defaulting to 0.

Now the idea is to slowly walk trough this list (to avoid getting banned from sites by flooding them with requests) while giving back a visual indication of what's going on. (without relying on ajax even)

 

So my idea was to have 2 iframes

1) A self refreshing page(via the meta tag and a ~30 sec delay or so) with the visual feedback a, list with the last 10 added values. And all results with count = 0

2) Hidden iframe initiating the process of walking trough the list like this

* SELECT blabla from blabla WHERE count = 0 AND limit = 1;

* Get the blabla and feed it to curl, and if the retrieved content was page could not be found or something update the count +1. If it was succesfull move it to another table.

* Sleep for a second and check in the database if there anything with count = 0 left, if so let it curl itself.

 

Good way to go about it, or is there a better way?

 

Share this post


Link to post
Share on other sites

I don't completely understand how you want it to work, but can't you use sleep to make sure you wait at least a fixed amount of seconds between each request?

Share this post


Link to post
Share on other sites

I don't completely understand how you want it to work, but can't you use sleep to make sure you wait at least a fixed amount of seconds between each request?

Sleep for a second and check in the database if there anything with count = 0 left, if so let it curl itself."

So yeah, that was the plan, can't do that on the page itself as it would result in a slow loading page refreshing all the time.

 

I forgot to mention btw that an hourly cron job is adding the links to the database and I planned on letting that take care of the links where the count is greater than 1. Up till a yet to determine number where the link will switch its status to "impossible" or something as it might just be a lost cause if isn't up in say ~2 days.

Share this post


Link to post
Share on other sites

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.