Use multiple proxies to pull content via php

[arianhojat]

I would like to do something like below but am afraid the hosts i am getting info from will ban requests from my url.

 

$url = "www.myspace.com/". getUserName($userID);

$content=file_get_contents($url ,FALSE,NULL);

//parse html and store some parts in a database.

 

Do you think a site like myspace will give a poop about lil' old me and ban requests from my server?

The requests are user initated, and will only parse their webpage. So im not doing anything evil here.

Just worried when I run requests everynight, they will see a flood of requests from 1 server and ban the IP.

 

Is there anyway to get http information through various trustable proxies and what functions should i check out?

[BlueSkyIS]

Depending on the number of requests, you might slow them down by running one per crontab job, or alternatively using sleep($number_of_seconds) in between requests.

 

 

[arianhojat]

if I have many thousands of users sign up in 1st day my site goes live, I am worried then spreading them around a day will still be obvious.

[BlueSkyIS]

yes, I would be concerned about that.

 

If you need to use proxies, you'll want to look into curl functions which provide this ability.

 

 

[arianhojat]

I was looking at curl a while back to do paypal stuff, it was a tiny bit confusing.

 

Any simple code examples of curl getting http info through a proxy?

my best guess is use the proxy websites form url to curl and supply it the info of the url you want to visit and POST.

must be a more elegant way of gettign it through the proxy without using the form on the page.

 

 

Also what proxies could I trust to return reliable information?

is there a list of reliable ones?

i use hidmyass_com haha and see they supply a daily list of proxies, i could always use that as my list to randomly pick from.