Jump to content

Retrieving information from other websites?


php_b34st

Recommended Posts

Is this possible? I would like to make a clan memberlist for runescape. The idea is that all the members of my clan will be listed with their runescape statistics but to do this i will need to access runescape's hi scores and then diplay them on my site. Can this be done with php? if so an someone give me some pointers in starting this script?

Thanks in advance
Link to comment
Share on other sites

[quote author=Caesar link=topic=101431.msg401470#msg401470 date=1153524031]
I will not give you code....because I think that defeats the learning process, and it undermines the work other people put in to figure it out...but I will message you with a hint/advice on what direction to take.
[/quote]

I agree just handing out code does defeat the learning pricess, as for hints/advice could you post here since the messaging system is disaled?
Link to comment
Share on other sites

2 main approaches:

If its XML markup, DOM can be of great use.

If its not, regular expressions.

Of course, thats for prossesing the data. Getting the data is the EASY part. (you just grab the whole stinkin HTML page or RSS feed. If you dont know how to do this, read the PHP manual, specifically, functions like fopen() and file())


As for the semantics of grabing speicifically runescape highscores, there are things you must consider:
- How do i view only the entrys i want?
- Can i do this from within a PHP script? (eg: What if it only accepts local POST requests to search for users?)
- Do i cache the data on my machine to reduce ammount of prossesing required?
if so, how do i 'decide' when to update the data?

There is of course more, but that should get you started.
Link to comment
Share on other sites

Ok, so i looked up the file() and preg_match() functions. I came up with the following code which seems to work untill i looked into it a bit closer:

[code]<?php

$user = 'zezima';

function RSstats($url)
{
$rssstring = file_get_contents($url);
preg_match_all('#<td align="right">(.*?)</td>#s',$rssstring,$stat);

echo '<table border="1"><tr><td>';

    echo $stat[1][0].'</td><td>'
.$stat[1][1].'</td><td>'
.$stat[1][2].'</td><td>'
.$stat[1][3].'</td><td></tr>'
.'<tr><td>Overall</td><td>'
.$stat[1][4].'</td><td>'
.$stat[1][5].'</td><td>'
.$stat[1][6].'</td><td></tr>'
.'<tr><td>Attack</td><td>'
.$stat[1][6].'</td><td>'
.$stat[1][9].'</td><td>'
.$stat[1][10].'</td><td></tr>'
.'<tr><td>Defense</td><td>'
.$stat[1][12].'</td><td>'
.$stat[1][13].'</td><td>'
.$stat[1][14].'</td><td></tr>'
.'<tr><td>Strength</td><td>'
.$stat[1][16].'</td><td>'
.$stat[1][17].'</td><td>'
.$stat[1][18].'</td><td></tr>'
.'<tr><td>Hitpoints</td><td>'
.$stat[1][20].'</td><td>'
.$stat[1][21].'</td><td>'
.$stat[1][22].'</td><td></tr>'
.'<tr><td>Ranged</td><td>'
.$stat[1][24].'</td><td>'
.$stat[1][25].'</td><td>'
.$stat[1][26].'</td><td></tr>'
.'<tr><td>Prayer</td><td>'
.$stat[1][28].'</td><td>'
.$stat[1][29].'</td><td>'
.$stat[1][30].'</td><td></tr>'
.'<tr><td>Magic</td><td>'
.$stat[1][32].'</td><td>'
.$stat[1][33].'</td><td>'
.$stat[1][34].'</td><td></tr>'
.'<tr><td>Cooking</td><td>'
.$stat[1][36].'</td><td>'
.$stat[1][37].'</td><td>'
.$stat[1][38].'</td><td></tr>'
.'<tr><td>Woodcutting</td><td>'
.$stat[1][40].'</td><td>'
.$stat[1][41].'</td><td>'
.$stat[1][42].'</td><td></tr>'
.'<tr><td>Fletching</td><td>'
.$stat[1][44].'</td><td>'
.$stat[1][45].'</td><td>'
.$stat[1][46].'</td><td></tr>'
.'<tr><td>Fishing</td><td>'
.$stat[1][48].'</td><td>'
.$stat[1][49].'</td><td>'
.$stat[1][50].'</td><td></tr>'
.'<tr><td>Firemaking</td><td>'
.$stat[1][52].'</td><td>'
.$stat[1][53].'</td><td>'
.$stat[1][54].'</td><td></tr>'
.'<tr><td>Crafting</td><td>'
.$stat[1][56].'</td><td>'
.$stat[1][57].'</td><td>'
.$stat[1][58].'</td><td></tr>'
.'<tr><td>Smithing</td><td>'
.$stat[1][60].'</td><td>'
.$stat[1][61].'</td><td>'
.$stat[1][62].'</td><td></tr>'
.'<tr><td>Mining</td><td>'
.$stat[1][64].'</td><td>'
.$stat[1][65].'</td><td>'
.$stat[1][66].'</td><td></tr>'
.'<tr><td>Herblore</td><td>'
.$stat[1][68].'</td><td>'
.$stat[1][69].'</td><td>'
.$stat[1][70].'</td><td></tr>'
.'<tr><td>Agility</td><td>'
.$stat[1][72].'</td><td>'
.$stat[1][73].'</td><td>'
.$stat[1][74].'</td><td></tr>'
.'<tr><td>Thieving</td><td>'
.$stat[1][76].'</td><td>'
.$stat[1][77].'</td><td>'
.$stat[1][78].'</td><td></tr>'
.'<tr><td>Slayer</td><td>'
.$stat[1][80].'</td><td>'
.$stat[1][81].'</td><td>'
.$stat[1][82].'</td><td></tr>'
.'<tr><td>Farming</td><td>'
.$stat[1][84].'</td><td>'
.$stat[1][85].'</td><td>'
.$stat[1][86].'</td><td></tr>'
.'<tr><td>Runecraft</td><td>'
.$stat[1][88].'</td><td>'
.$stat[1][89].'</td><td>'
.$stat[1][90].'</td><td></tr>'
.'<tr><td>Construction</td><td>'
.$stat[1][92].'</td><td>'
.$stat[1][93].'</td><td>'
.$stat[1][94].'</td><td></tr></table>';
}

RSstats('http://hiscore.runescape.com/lang/en/aff/runescape/hiscorepersonal.ws?user1=' . $user);


?>[/code]

If you where to test that code it would display a table with zezima's statistics in it, but the problem becomes apparent when you change the user. zezima has all of his stats ranked so there is no problem with getting his info, but what if someone is unranked? for example change user to andey and the table starts to go funny at farming because both his farming and construction stats are not ranked.

Is there a way i can fix this? any help would be greatly appreciated
Link to comment
Share on other sites

I have worked around the previous problem and now have the following code which does exactly what i wanted. the problem now is that the code is very long and it takes some time for the page to load. Is there a shorter way to do this so that the page loads quicker?

new code:
[code]<?php

$user = 'zezima';
$data = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/hiscorepersonal.ws?user1=' . $user);

if (preg_match('/\bdoes not feature\b/i',$data))
{
echo $user.' does not feature in the hiscores. You have to be in the top 1 million (for any skill) and have a minimum skill level of 30';
exit;
}
else
{
echo '<h1>Runescape Scores for '.$user.'</h1>';

echo '<table border="1">
        <tr>
  <td>Skill</td>
  <td>Rank</td>
  <td>Level</td>
  <td>XP</td>
</tr>
<tr>';

if (preg_match('/\btable=0\b/i',$data))
{
$overall = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=0&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $overall, $stat);
echo '<td>Overall</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Overall</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=1\b/i',$data))
{
$attack = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=1&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $attack, $stat);
echo '<td>Attack</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Attack</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=2\b/i',$data))
{
$defense = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=2&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $defense, $stat);
echo '<td>Defense</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Defense</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=3\b/i',$data))
{
$strength = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=3&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $strength, $stat);
echo '<td>Strength</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Strength</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=4\b/i',$data))
{
$hp = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=4&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $hp, $stat);
echo '<td>Hitpoints</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Hitpoints</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=5\b/i',$data))
{
$range = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=5&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $range, $stat);
echo '<td>Ranged</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Ranged</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=6\b/i',$data))
{
$prayer = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=6&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $prayer, $stat);
echo '<td>Prayer</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Prayer</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=7\b/i',$data))
{
$mage = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=7&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $mage, $stat);
echo '<td>Magic</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Magic</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=8\b/i',$data))
{
$cooking = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=8&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $cooking, $stat);
echo '<td>Cooking</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Cooking</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=9\b/i',$data))
{
$wc = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=9&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $wc, $stat);
echo '<td>Woodcutting</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Woodcutting</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=10\b/i',$data))
{
$fletch = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=10&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $fletch, $stat);
echo '<td>Fletching</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Fletching</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=11\b/i',$data))
{
$fish = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=11&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $fish, $stat);
echo '<td>Fishing</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Fishing</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=12\b/i',$data))
{
$fire = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=12&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $fire, $stat);
echo '<td>Firemaking</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Firemaking</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=13\b/i',$data))
{
$craft = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=13&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $craft, $stat);
echo '<td>Crafting</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Crafting</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=14\b/i',$data))
{
$smith = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=14&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $smith, $stat);
echo '<td>Smithing</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Smithing</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=15\b/i',$data))
{
$mining = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=15&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $mining, $stat);
echo '<td>Mining</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Mining</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=16\b/i',$data))
{
$herblore = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=16&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $herblore, $stat);
echo '<td>Herblore</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Herblore</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=17\b/i',$data))
{
$agility = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=17&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $agility, $stat);
echo '<td>Agility</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Agility</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=18\b/i',$data))
{
$thieving = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=18&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $thieving, $stat);
echo '<td>Thieving</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Thieving</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=19\b/i',$data))
{
$slayer = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=19&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $slayer, $stat);
echo '<td>Slayer</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Slayer</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=20\b/i',$data))
{
$farm = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=20&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $farm, $stat);
echo '<td>Farming</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Farming</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=21\b/i',$data))
{
$rune = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=21&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $rune, $stat);
echo '<td>Runecraft</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Runecraft</td><td colspan="3" align="right">Not Ranked</td></tr>';
}

if (preg_match('/\btable=23\b/i',$data))
{
$con = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=23&user=' . $user);
preg_match_all('#AA0022">(.*?)</font>#s', $con, $stat);
echo '<td>Construction</td><td>'
    .$stat[1][0].'</td><td>'
    .$stat[1][2].'</td><td>'
    .$stat[1][3].'</td></tr>';

}
else
{
echo '<td>Construction</td><td colspan="3" align="right">Not Ranked</td></tr>';
}
}

?>[/code]
Link to comment
Share on other sites

Whoah! first things first: use a loop!

Secont thing: Can't you jsut grab all the data from a single page, use a regular expression to grab what you want, format it into an array, and then display it? (as opposed to grabbing a new file for every diffrent skill)

Edit: Oh, and good job  :) I wish I learn that fast  :o
Link to comment
Share on other sites

I am not so sure on making loops and it was prooving difficult for the above code so i decided to start in an easier place - the main memberlist. the following code will grab the overall score for each username in my db but again it takes some time to load, i assume this is because of the way i am grabbing the data?

[code]<?php
include('db.php');

$query = "SELECT * FROM users";
$result = mysql_query($query) or die("Could not query: " . mysql_error());

$info = array();
$i = 0;


while ($row = mysql_fetch_assoc($result)) {
    $info[$i]['username'] = $row['username'];
    $overall = file_get_contents('http://hiscore.runescape.com/lang/en/aff/runescape/overall.ws?table=0&user=' . $row['username']);
    preg_match_all('#AA0022">(.*?)</font>#s', $overall, $stat);
    $info[$i]['overall'] = $stat[1][2];
    $i++;
   
}

function desc($a, $b)
{
  if($a['overall'] == $b['rank'])
      return 0;
  else
      return ($a['overall'] > $b['rank']) ? -1 : 1;
}

usort($info, 'desc');

echo '<table align="center" border="1">
        <tr>
            <th>Rank</th>
            <th>Username</th>
            <th>Overall</th>
        </tr>';
$i = 0;
foreach ($info as $r => $value) {
$username = $info[$r]['username'];
$rank = $i + 1;
    echo '
        <tr>
            <td>'.$rank .'</td>
    <td>'. $username . '</td>
    <td>'. $info[$r]['overall']. '</td>
        </tr>';  
    $i++;
}

?>[/code]

[quote author=joehaley link=topic=101431.msg401669#msg401669 date=1153578969]
Secont thing: Can't you jsut grab all the data from a single page, use a regular expression to grab what you want, format it into an array, and then display it? (as opposed to grabbing a new file for every diffrent skill)
[/quote]

Any suggestion on how i would do this?
Link to comment
Share on other sites

It takes time to load as you must grab the entire page, disect it, do what ya want with it, when output to the user.

You could always make a cron job to automatically grab data and update it daily, storing only what you need in a local database.

Or, if you dont know about crons, randomly update the database when the page is viewd (thus causing only ocasional long load times. maybe toss in a manual reload button too)
Link to comment
Share on other sites

The idea was that the table would be constantly up to date, but if this cannot be done without the delay i may have to look into storing the data in a database. I have heard of cron jons but unfortunately i am using a windows server. Thanks for your help.
Link to comment
Share on other sites

This thread is more than a year old. Please don't revive it unless you have something important to add.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.