[SOLVED] URL Next, URL Next, URL Next == cURL?

[Modernvox]

Good evening fellow grasshoppers

 

I am opening multiple URL's using cURL. Since i am using cURL I am wondering how i can open the second URL after opening the first?

 

I open the first one and check for appropriate links. Now i want to open those links.

How do i open another URL/Link if I already have cURL pointing to a URL?

 

Do I just retype curl_setopt($curl, CURLOPT_URL, $url);  and just change the current URL $curlResults = curlURL("http://southcoast.craigslist.org/sss/"); 

 

Would i be better off using C++ to create things such as auto-opening url's and grabbing info or is PHP fine?

 

 

So how do i open the first link in a vertical list of links?? Please??

Here's what the links look like

 

Links:

 

/pts/1440335603.html

/cto/1440337386.html

/for/1440337113.html

/boa/1440333500.html

/wan/1440333103.html

/msg/1440331732.html

/fuo/1440328124.html

/hsh/1440329361.html

/zip/1440326866.html

/boa/1440325266.html

/tag/1433958179.html

/cto/1440319544.html

/cto/1440318051.html

/atq/1440306915.html

/grd/1440312318.html

/cto/1440309994.html

/mcy/1440309840.html

/cto/1440309191.html

/vgm/1440308075.html

/ctd/1440307983.html

/cto/1440305846.html

/fuo/1440258609.html

/grd/1440279567.html

/vgm/1440302253.html

 

 <?php  
    function curlURL($url) {  
        $curl = curl_init();  
        curl_setopt($curl, CURLOPT_URL, $url);  
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);  
        curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2');  
        $output = curl_exec($curl);  
            return $output;  
    }  
     
   $curlResults = curlURL("http://southcoast.craigslist.org/sss/");  
   $pattern = '#<a href="(/[a-z]{3}/\d{10}\.html)">#';
   preg_match_all( $pattern, $curlResults, $matches);

echo "<pre>\n";
echo "Links:\n\n";
foreach ($matches[1] as $link):
   echo "\t" . '<a href="' . $link . '" target="_BLANK">' . $link . '</a>' . "\n";
endforeach;
echo '</pre>';

?> 

[mrMarcus]

you are looking to use cURL against each of those links simultaneously?  i don't know that craigslist with to too happy with that.

 

sorry if i'm off, but i'm not quite following your intentions with these urls.

[Modernvox]

you are looking to use cURL against each of those links simultaneously?  i don't know that craigslist with to too happy with that.

 

sorry if i'm off, but i'm not quite following your intentions with these urls.

 

Yeah. I want to open the first link check for email address then close and open second etc, etc..

[mrMarcus]

FYI, they (craigslist) will ban your ip (which in this case would be your ip address associated with your host).  matter of fact, i'm quite certain they have laser beams that will pulverize you from the heavens for doing such a thing.

 

but who am i stand in your way.

 

i would have to say, the only real way i can see doing this (keep in mind, i haven't put a whole lot of brainpower into it), is to loop each URL using foreach(), and executing your harvesting code within that same loop .. output your results to an array (or file/database), and there you go .. making sure to close each cURL connection within the loop: curl_close($curl);  i'm not sure if cURL allows for concurrent connections as i've never needed such a thing yet.

 

i don't know how long a process this might be, but i'm quite sure you're just looking for results and can spare a few minutes.

[Modernvox]

FYI, they (craigslist) will ban your ip (which in this case would be your ip address associated with your host).  matter of fact, i'm quite certain they have laser beams that will pulverize you from the heavens for doing such a thing.

 

but who am i stand in your way.

 

i would have to say, the only real way i can see doing this (keep in mind, i haven't put a whole lot of brainpower into it), is to loop each URL using foreach(), and executing your harvesting code within that same loop .. output your results to an array (or file/database), and there you go .. making sure to close each cURL connection within the loop: curl_close($curl);  i'm not sure if cURL allows for concurrent connections as i've never needed such a thing yet.

 

i don't know how long a process this might be, but i'm quite sure you're just looking for results and can spare a few minutes.

 

lol. Thanks dude

[Modernvox]

What would the foreach loop look like in this case?

 

 

[Modernvox]

I'll wait.....