Retrieving information from other websites?

[php_b34st]

Is this possible? I would like to make a clan memberlist for runescape. The idea is that all the members of my clan will be listed with their runescape statistics but to do this i will need to access runescape's hi scores and then diplay them on my site. Can this be done with php? if so an someone give me some pointers in starting this script?

Thanks in advance

[Caesar]

Yes this is possible.

And was it you that asked this question before?  :)

[php_b34st]

Do you know how it is done? No I have not asked this question before

[Caesar]

I will not give you code....because I think that defeats the learning process, and it undermines the work other people put in to figure it out...but I will message you with a hint/advice on what direction to take.

[Caesar]

Looks like the private messaging has been disabled. :-P

[php_b34st]

[quote author=Caesar link=topic=101431.msg401470#msg401470 date=1153524031]
I will not give you code....because I think that defeats the learning process, and it undermines the work other people put in to figure it out...but I will message you with a hint/advice on what direction to take.
[/quote]

I agree just handing out code does defeat the learning pricess, as for hints/advice could you post here since the messaging system is disaled?

[Joe Haley]

2 main approaches:

If its XML markup, DOM can be of great use.

If its not, regular expressions.

Of course, thats for prossesing the data. Getting the data is the EASY part. (you just grab the whole stinkin HTML page or RSS feed. If you dont know how to do this, read the PHP manual, specifically, functions like fopen() and file())


As for the semantics of grabing speicifically runescape highscores, there are things you must consider:
- How do i view only the entrys i want?
- Can i do this from within a PHP script? (eg: What if it only accepts local POST requests to search for users?)
- Do i cache the data on my machine to reduce ammount of prossesing required?
if so, how do i 'decide' when to update the data?

There is of course more, but that should get you started.

[php_b34st]

Ok thanks that should get me started, I will look in the manual now

[php_b34st]

Ok, so i looked up the file() and preg_match() functions. I came up with the following code which seems to work untill i looked into it a bit closer:

[code]<?php

$user = 'zezima';

function RSstats($url)
{
$rssstring = file_get_contents($url);
preg_match_all('#<td align="right">(.*?)</td>#s',$rssstring,$stat);

echo '<table border="1"><tr><td>';

    echo $stat[1][0].'</td><td>'
.$stat[1][1].'</td><td>'
.$stat[1][2].'</td><td>'
.$stat[1][3].'</td><td></tr>'
.'<tr><td>Overall</td><td>'
.$stat[1][4].'</td><td>'
.$stat[1][5].'</td><td>'
.$stat[1][6].'</td><td></tr>'
.'<tr><td>Attack</td><td>'
.$stat[1][6].'</td><td>'
.$stat[1][9].'</td><td>'
.$stat[1][10].'</td><td></tr>'
.'<tr><td>Defense</td><td>'
.$stat[1][12].'</td><td>'
.$stat[1][13].'</td><td>'
.$stat[1][14].'</td><td></tr>'
.'<tr><td>Strength</td><td>'
.$stat[1][16].'</td><td>'
.$stat[1][17].'</td><td>'
.$stat[1][18].'</td><td></tr>'
.'<tr><td>Hitpoints</td><td>'
.$stat[1][20].'</td><td>'
.$stat[1][21].'</td><td>'
.$stat[1][22].'</td><td></tr>'
.'<tr><td>Ranged</td><td>'
.$stat[1][24].'</td><td>'
.$stat[1][25].'</td><td>'
.$stat[1][26].'</td><td></tr>'
.'<tr><td>Prayer</td><td>'
.$stat[1][28].'</td><td>'
.$stat[1][29].'</td><td>'
.$stat[1][30].'</td><td></tr>'
.'<tr><td>Magic</td><td>'
.$stat[1][32].'</td><td>'
.$stat[1][33].'</td><td>'
.$stat[1][34].'</td><td></tr>'
.'<tr><td>Cooking</td><td>'
.$stat[1][36].'</td><td>'
.$stat[1][37].'</td><td>'
.$stat[1][38].'</td><td></tr>'
.'<tr><td>Woodcutting</td><td>'
.$stat[1][40].'</td><td>'
.$stat[1][41].'</td><td>'
.$stat[1][42].'</td><td></tr>'
.'<tr><td>Fletching</td><td>'
.$stat[1][44].'</td><td>'
.$stat[1][45].'</td><td>'
.$stat[1][46].'</td><td></tr>'
.'<tr><td>Fishing</td><td>'
.$stat[1][48].'</td><td>'
.$stat[1][49].'</td><td>'
.$stat[1][50].'</td><td></tr>'
.'<tr><td>Firemaking</td><td>'
.$stat[1][52].'</td><td>'
.$stat[1][53].'</td><td>'
.$stat[1][54].'</td><td></tr>'
.'<tr><td>Crafting</td><td>'
.$stat[1][56].'</td><td>'
.$stat[1][57].'</td><td>'
.$stat[1][58].'</td><td></tr>'
.'<tr><td>Smithing</td><td>'
.$stat[1][60].'</td><td>'
.$stat[1][61].'</td><td>'
.$stat[1][62].'</td><td></tr>'
.'<tr><td>Mining</td><td>'
.$stat[1][64].'</td><td>'
.$stat[1][65].'</td><td>'
.$stat[1][66].'</td><td></tr>'
.'<tr><td>Herblore</td><td>'
.$stat[1][68].'</td><td>'
.$stat[1][69].'</td><td>'
.$stat[1][70].'</td><td></tr>'
.'<tr><td>Agility</td><td>'
.$stat[1][72].'</td><td>'
.$stat[1][73].'</td><td>'
.$stat[1][74].'</td><td></tr>'
.'<tr><td>Thieving</td><td>'
.$stat[1][76].'</td><td>'
.$stat[1][77].'</td><td>'
.$stat[1][78].'</td><td></tr>'
.'<tr><td>Slayer</td><td>'
.$stat[1][80].'</td><td>'
.$stat[1][81].'</td><td>'
.$stat[1][82].'</td><td></tr>'
.'<tr><td>Farming</td><td>'
.$stat[1][84].'</td><td>'
.$stat[1][85].'</td><td>'
.$stat[1][86].'</td><td></tr>'
.'<tr><td>Runecraft</td><td>'
.$stat[1][88].'</td><td>'
.$stat[1][89].'</td><td>'
.$stat[1][90].'</td><td></tr>'
.'<tr><td>Construction</td><td>'
.$stat[1][92].'</td><td>'
.$stat[1][93].'</td><td>'
.$stat[1][94].'</td><td></tr></table>';
}

RSstats('http://hiscore.runescape.com/lang/en/aff/runescape/hiscorepersonal.ws?user1=' . $user);


?>[/code]

If you where to test that code it would display a table with zezima's statistics in it, but the problem becomes apparent when you change the user. zezima has all of his stats ranked so there is no problem with getting his info, but what if someone is unranked? for example change user to andey and the table starts to go funny at farming because both his farming and construction stats are not ranked.

Is there a way i can fix this? any help would be greatly appreciated

[php_b34st]

I have worked around the previous problem and now have the following code which does exactly what i wanted. the problem now is that the code is very long and it takes some time for the page to load. Is there a shorter way to do this so that the page loads quicker?

new code:
[code]<?php

$user = 'zezima';
$data = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/hiscorepersonal.ws?user1=' . $user);

if (preg_match('/\bdoes not feature\b/i',$data))
{
echo $user.' does not feature in the hiscores. You have to be in the top 1 million (for any skill) and have a minimum skill level of 30';
exit;
}
else
{
echo '<h1>Runescape Scores for '.$user.'</h1>';

echo '<table border="1">
        <tr>
  <td>Skill</td>
  <td>Rank</td>
  <td>Level</td>
  <td>XP</td>
</tr>
<tr>';

if (preg_match('/\btable=0\b/i',$data))
{
$overall = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=0&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $overall, $stat);
echo '<td>Overall</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Overall</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=1\b/i',$data))
{
$attack = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=1&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $attack, $stat);
echo '<td>Attack</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Attack</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=2\b/i',$data))
{
$defense = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=2&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $defense, $stat);
echo '<td>Defense</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Defense</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=3\b/i',$data))
{
$strength = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=3&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $strength, $stat);
echo '<td>Strength</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Strength</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=4\b/i',$data))
{
$hp = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=4&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $hp, $stat);
echo '<td>Hitpoints</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Hitpoints</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=5\b/i',$data))
{
$range = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=5&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $range, $stat);
echo '<td>Ranged</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Ranged</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=6\b/i',$data))
{
$prayer = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=6&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $prayer, $stat);
echo '<td>Prayer</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Prayer</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=7\b/i',$data))
{
$mage = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=7&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $mage, $stat);
echo '<td>Magic</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Magic</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=8\b/i',$data))
{
$cooking = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=8&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $cooking, $stat);
echo '<td>Cooking</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Cooking</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=9\b/i',$data))
{
$wc = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=9&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $wc, $stat);
echo '<td>Woodcutting</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Woodcutting</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=10\b/i',$data))
{
$fletch = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=10&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $fletch, $stat);
echo '<td>Fletching</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Fletching</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=11\b/i',$data))
{
$fish = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=11&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $fish, $stat);
echo '<td>Fishing</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Fishing</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=12\b/i',$data))
{
$fire = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=12&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $fire, $stat);
echo '<td>Firemaking</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Firemaking</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=13\b/i',$data))
{
$craft = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=13&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $craft, $stat);
echo '<td>Crafting</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Crafting</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=14\b/i',$data))
{
$smith = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=14&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $smith, $stat);
echo '<td>Smithing</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Smithing</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=15\b/i',$data))
{
$mining = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=15&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $mining, $stat);
echo '<td>Mining</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Mining</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=16\b/i',$data))
{
$herblore = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=16&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $herblore, $stat);
echo '<td>Herblore</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Herblore</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=17\b/i',$data))
{
$agility = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=17&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $agility, $stat);
echo '<td>Agility</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Agility</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=18\b/i',$data))
{
$thieving = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=18&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $thieving, $stat);
echo '<td>Thieving</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Thieving</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=19\b/i',$data))
{
$slayer = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=19&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $slayer, $stat);
echo '<td>Slayer</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Slayer</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=20\b/i',$data))
{
$farm = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=20&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $farm, $stat);
echo '<td>Farming</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Farming</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=21\b/i',$data))
{
$rune = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=21&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $rune, $stat);
echo '<td>Runecraft</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Runecraft</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=23\b/i',$data))
{
$con = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=23&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $con, $stat);
echo '<td>Construction</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Construction</td><td colspan="3" align="right">Not Ranked</td></tr>';
}
}

?>[/code]

[Joe Haley]

Whoah! first things first: use a loop!

Secont thing: Can't you jsut grab all the data from a single page, use a regular expression to grab what you want, format it into an array, and then display it? (as opposed to grabbing a new file for every diffrent skill)

Edit: Oh, and good job  :) I wish I learn that fast  :o

[php_b34st]

Thanks again for the advice i will look into putting into a loop but the problem was the table changes for each skill, i did initially try to grab all the data from one page but had problems when the user wasnt ranked.

[php_b34st]

I am not so sure on making loops and it was prooving difficult for the above code so i decided to start in an easier place - the main memberlist. the following code will grab the overall score for each username in my db but again it takes some time to load, i assume this is because of the way i am grabbing the data?

[code]<?php
include('db.php');

$query = "SELECT * FROM users";
$result = mysql_query($query) or die("Could not query: " . mysql_error());

$info = array();
$i = 0;


while ($row = mysql_fetch_assoc($result)) {
    $info[$i]['username'] = $row['username'];
    $overall = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=0&user=' . $row['username']);
    preg_match_all('#AA0022">(.*?)</font>#s', $overall, $stat);
    $info[$i]['overall'] = $stat[1][2];
    $i++;
   
}

function desc($a, $b)
{
  if($a['overall'] == $b['rank'])
      return 0;
  else
      return ($a['overall'] > $b['rank']) ? -1 : 1;
}

usort($info, 'desc');

echo '<table align="center" border="1">
        <tr>
            <th>Rank</th>
            <th>Username</th>
            <th>Overall</th>
        </tr>';
$i = 0;
foreach ($info as $r => $value) {
$username = $info[$r]['username'];
$rank = $i + 1;
    echo '
        <tr>
            <td>'.$rank .'</td>
    <td>'. $username . '</td>
    <td>'. $info[$r]['overall']. '</td>
        </tr>';  
    $i++;
}

?>[/code]

[quote author=joehaley link=topic=101431.msg401669#msg401669 date=1153578969]
Secont thing: Can't you jsut grab all the data from a single page, use a regular expression to grab what you want, format it into an array, and then display it? (as opposed to grabbing a new file for every diffrent skill)
[/quote]

Any suggestion on how i would do this?

[Joe Haley]

It takes time to load as you must grab the entire page, disect it, do what ya want with it, when output to the user.

You could always make a cron job to automatically grab data and update it daily, storing only what you need in a local database.

Or, if you dont know about crons, randomly update the database when the page is viewd (thus causing only ocasional long load times. maybe toss in a manual reload button too)

[php_b34st]

The idea was that the table would be constantly up to date, but if this cannot be done without the delay i may have to look into storing the data in a database. I have heard of cron jons but unfortunately i am using a windows server. Thanks for your help.