How to make my crawler go to the next page?

[swampone]

Im stuck. Im building a crawler to extract video links and images from different sites using pre match all and cURL. So far Im able to get all the information I want from one page. Problem is I cant get it to go through page by page extracting the data i want.

[cags]

What other pages do you wish to call? You could crawl recursively if you are attempting to create your own web crawler, but this will potentially never end. To call recursively you will have to extract all the links from the page (either using preg_match or some other function) then create a foreach() loop to loop through each of those files calling the same function your currently in.

 

A quick pseudo example.

 

function crawl( $url ) {
  $file = file_get_contents( $url );

  if ( preg_match ( $some_valid_url_pattern, $file, $output ) ) {
    foreach ( $output as $url ) {
      crawl( $url );
    }
  }
}

[swampone]

You know when you are looking at a site like you and you search for a video, well there is usually pages of videos that come up. I'm only able to get the videos off the first page. I need to be able to get all the videos.