[SOLVED] Grab Data from another website.

[`Karl]

Well, I need to grab data from another website, which isn't my own.

 

I've already contacted the website to see if I could have permission to screen scrape, however, they declined so I'm looking for an alternative way.

 

An example of a page I want to get information from can be seen here:

http://services.runescape.com/m=itemdb_rs/Abyssal_whip/viewitem.ws?obj=4151

 

The information I need is:

3.4m, 3.6m and 3.8m

 

Any input would be appreciated.

[sKunKbad]

You would want to use file_get_contents or similar function, and then search for:

 

<span>
<b>Minimum price:</b> 3.4m  
</span>
<span class="spaced_span">
<b>Market price:</b> 3.6m  
</span>
<span>
<b>Maximum price:</b> 3.8m  
</span>

 

replacing the numbers with regular expressions for float type numbers, which would be part of a callback to retrieve that actual data. I don't know if this is the most efficient way of doing it, but it's all I could think of.

 

[`Karl]

Mmm, I'll have a look in to it.

 

Edit: Can't figure it out >.<

[`Karl]

Anyone have any other ideas?

[`Karl]

Anyone?

[sKunKbad]

Without studying a bit on regular expressions, your not likely to figure it out, but the function that I would try to use on the data obtained from file_get_contents would be preg_match. See the preg_match page in the docs to see usage, and then go over to the regular expressions forum to see if you are doing it right.

[`Karl]

Any way you can push me in the right direction?

 

<?php 
$data = file_get_contents('http://services.runescape.com/m=itemdb_rs/Abyssal_whip/viewitem.ws?obj=4151');
$regex = '/Minimum Price:<\/b> (.+?)<\/span>/';
preg_match($regex,$data,$match);
var_dump($match);
echo $match[1];
?>

 

Returns:

array(0) { }

[`Karl]

Okay, so I had a mess around and got it working, however, when I tried to change it so that a user must use a form and type in an item number, it just comes back blank.

 

<?php
$itemnumber = $_GET['itemnumber'];
$lookup = $_POST['lookup'];
$minimum = explode('<b>Minimum price:</b> ',$content);
$minimum2 = explode('</span>',$minimum[1]);
$minimumprice = $minimum2[0];

$current = explode('<b>Market price:</b> ',$content);
$current2 = explode('</span>',$current[1]);
$currentprice = $current2[0];

$maximum = explode('<b>Maximum price:</b> ',$content);
$maximum2 = explode('</span>',$maximum[1]);
$maximumprice = $maximum2[0];

?> 
<form action="" method="POST">
<input type="text" name="itemnumber" />

<input type="submit" value="Lookup" name="lookup" />
</form>
<?php
if(isset($_POST['lookup'])) {
$content = file_get_contents('http://services.runescape.com/m=itemdb_rs/viewitem.ws?obj=' . $itemnumber);
echo 'Minimum Price: ' . $minimumprice;
echo '<br>';
echo 'Market Price: ' . $currentprice;
echo '<br>';
echo 'Maximum Price: ' . $maximumprice;
unset($content,$minimum,$minimum2,$current,$current2,$maximum,$maximum2);
}
?>

 

If a user puts in 4151 in to the box, which is a valid item. Then the Minimum Price all return blank, when they should in fact return the prices.

 

Whereas this works perfectly and returns the data:

<?php
$content = file_get_contents('http://services.runescape.com/m=itemdb_rs/viewitem.ws?obj=4151');
$minimum = explode('<b>Minimum price:</b> ',$content);
$minimum2 = explode('</span>',$minimum[1]);
$minimumprice = $minimum2[0];

$current = explode('<b>Market price:</b> ',$content);
$current2 = explode('</span>',$current[1]);
$currentprice = $current2[0];

$maximum = explode('<b>Maximum price:</b> ',$content);
$maximum2 = explode('</span>',$maximum[1]);
$maximumprice = $maximum2[0];

echo 'Minimum Price: ' . $minimumprice;
echo '<br>';
echo 'Market Price: ' . $currentprice;
echo '<br>';
echo 'Maximum Price: ' . $maximumprice;
unset($content,$minimum,$minimum2,$current,$current2,$maximum,$maximum2);

?>

 

[trq]

Didn't you just say they declined to allow you to do this?

[`Karl]

Indeed I did, but there are other communities that do the same thing. My guess is they didn't ask and no one came up with another method to retrieve the data.

 

I was only declined as it would take up a lot of server resources on their end. If there's a method to make that server load smaller then they'd allow it. If you know of a method, please, enlighten me.

[trq]

Indeed I did, but there are other communities that do the same thing

 

Ahh, right, I guess that makes it ok then.

[`Karl]

Don't try to be sarcastic, the communities I'm talking about are recognised fansites by the creators of the game. Even websites which are not recognised use this method.

 

I'm also guessing by your response that you do not know of an alternative method.

[trq]

I'm also guessing by your response that you do not know of an alternative method.

 

There is no alternate method unless you get permission to get the data directly from there database.

[`Karl]

Any idea why my code returns blank when I can see nothing wrong with it?

[joel24]

an ethical error maybe?

[joel24]

you've got your PHP all jumbled up.. you're calling to fetch the page from the server after you try to read it...

<?php
if(isset($_POST['lookup'])) {

$itemnumber = $_GET['itemnumber'];
$content = file_get_contents('http://services.runescape.com/m=itemdb_rs/viewitem.ws?obj=' . $itemnumber);

$lookup = $_POST['lookup'];
$minimum = explode('<b>Minimum price:</b> ',$content);
$minimum2 = explode('</span>',$minimum[1]);
$minimumprice = $minimum2[0];

$current = explode('<b>Market price:</b> ',$content);
$current2 = explode('</span>',$current[1]);
$currentprice = $current2[0];

$maximum = explode('<b>Maximum price:</b> ',$content);
$maximum2 = explode('</span>',$maximum[1]);
$maximumprice = $maximum2[0];
}
?>
<form action="" method="POST">
<input type="text" name="itemnumber" />

<input type="submit" value="Lookup" name="lookup" />
</form>
<?php
if(isset($_POST['lookup'])) {
echo 'Minimum Price: ' . $minimumprice;
echo '<br>';
echo 'Market Price: ' . $currentprice;
echo '<br>';
echo 'Maximum Price: ' . $maximumprice;
unset($content,$minimum,$minimum2,$current,$current2,$maximum,$maximum2);
}
?>

[sKunKbad]

I'm also guessing by your response that you do not know of an alternative method.

 

If anyone would know an alternate method, it would be thorpe.

[oni-kun]

Any idea why my code returns blank when I can see nothing wrong with it?

 

Your code obviously is not ethical. Data theft is wrong, You could get the mob after you!

[corrupshun]

How is this unethical??

WOW