[SOLVED] Only Using Unique Values in "for()"

[JSHINER]

<?php
$seed = "http://www.site.com/page.html";
$data = file_get_contents($seed);
if (preg_match_all(/http:[^"\s']+/", $data, $links)) 
{
     for ($i=0;$i<count($links[0]);$i++)  // I think the problem is here?
     {

$data_b = file_get_contents('http://www.site.com/sub/'. $links[0][$i]);

if (preg_match_all(/http:[^"\s']+/", $data_b, $links_b)) 
{

@header("Content-type: text/plain");

     for ($i_b=0;$i_b<count($links_b[0]);$i_b++) 
     
     {
     
     echo $links_b[0][$i_b]. "\n";
     
     }
}
}
}

?>

 

The above works great EXCEPT that if there are two of the same links on the first page, it spiders them twice, which is using up double the resources. How can I get it to only use one of the two links?

[wildteen88]

Add each link you check into an array, call this array $checked_urls. Then every time you go to parse a url check whether that url has been parsed already using is_array. If its in the array skip to next url, else parse url.

[JSHINER]

Not sure I follow?

[wildteen88]

Abit like this:

<?php

$seed = "http://www.site.com/page.html";
$data = file_get_contents($seed);

$checked_links = array();

if (preg_match_all("/http:[^\"\s']+/", $data, $links))
{
    foreach($links[0] as $key => $link)
    {
        // check that the link has not been parsed already
        if(!in_array($link, $checked_links))
        {
            // link has not been parsed we'll add it to the checked links array
            $checked_links[] = $link;

            $data_b = file_get_contents('http://www.site.com/sub/'. $link);

            if (preg_match_all("/http:[^\"\s']+/", $data_b, $links_b))
            {
                @header("Content-type: text/plain");

                foreach($links_b[0] as $skey => $sub_link)
                {
                    // check that the sublink has not been parsed already
                    if(!in_array($sub_link, $checked_links))
                    {
                        // sublink has not been parsed we'll add it to the checked links array
                        $checked_links[] = $sub_link;

                        echo $sub_link . "\n";
                    }
                }
            }
        }
    }
}

?>

Notice: I changed your for loops to foreach loops. foreach loops are much easier to loop through arrays.

 

[JSHINER]

Perfect. Thank you!