Prasing a sites Database

[rotoxis]

I have found a tutorial what ive been looking for for Agers

 

 

The outcome should go to http://itemdb-rs.runescape.com/frontpage.ws Use the search feature and pull the Result for Magic

 

Now i found a tutorial first you have to add the Table Structure

 

CREATE TABLE IF NOT EXISTS `items` (
  `itemid` int(255) NOT NULL,
  `url` varchar(500) NOT NULL,
  `itemname` varchar(500) NOT NULL,
  `minprice` varchar(255) NOT NULL,
  `maxprice` varchar(255) NOT NULL,
  `midprice` varchar(255) NOT NULL,
  `updated` time NOT NULL,
  `7days` varchar(500) NOT NULL,
  `30days` varchar(500) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

 

 

Now for Import.php

 

<?php

$user = "pie_root";

$pass = "password"

$host = "localhost";

$conn = mysql_connect($host, $user, $pass);

mysql_select_db("pie_runescape", $conn) or die(mysql_error());


$i = 0;

while($i < 20000) {
ini_set('max_execution_time', -1);

	$file = @fopen("http://itemdb-rs.runescape.com/viewitem.ws?obj=" . $i, "rb");

	$contents = '';

	while(!feof($file)) {

		$contents .= fread($file, 8192);

	}

	fclose($file);
	$lines = explode("\n", $contents);

if($lines[207] == 'The item you were trying to view could not be found.') {
	echo $lines[207] . "<br />";
} else {

	//print_r($lines);

	$lines[225] = explode(" ", $lines[225]);

	$day = str_replace(array('class="rise">', "</span>", 'class="drop">', 'class="stay">'), "", $lines[225][3]);
	$lines[228] = explode(" ", $lines[228]);
	$days = str_replace(array('class="rise">', '</span>', 'class="drop">', 'class="stay">'), "", $lines[228][3]);
	$min = explode(" ", $lines[214]);

	$mid = explode(" ", $lines[217]);

	$max = explode(" ", $lines[220]);
	$itemname = addslashes($lines[205]);
	mysql_query("INSERT INTO `items` VALUES ('$i', 'http://itemdb-rs.runescape.com/viewitem.ws?obj=" . $i . "', '$itemname', '$min[2]', '$max[2]', '$mid[2]', 'NOW()', '$day', '$days')");
}
	$i++;
}
?>

 

 

WHEN I RAN THIS CODE ON MY WEBHOST 000WEBHOST I GOT IP BANNED FROM MY SITE AND IT DID NOT IMPUT ANYTHING INTO THE DATABASE

 

Now next is Index.php

 

<?php
$user = "user";

$pass = "password";

$host = "localhost";

$conn = mysql_connect($host, $user, $pass);

mysql_select_db("pie_runescape", $conn) or die(mysql_error());

if(isset($_GET['id'])) {
echo "<table border=\"1\" width=\"80%\">\n";
echo "<tr><th>ID</th><th>Name</th><th>Min Price</th><th>Mid Price</th><th>Max Price</th><th>7Days</th><th>30days</th></tr>\n";
$id = mysql_real_escape_string($_GET['id']);
$query = mysql_query("SELECT * FROM `items` WHERE `itemid` = '$id'");
while($row = mysql_fetch_array($query)) {
	echo "<tr><td>" . $row['itemid'] . "</td><td>" . $row['itemname'] . "</td><td>" . $row['minprice'] . "</td><td>" . $row['midprice'] . "</td><td>" . $row['maxprice'] . "</td><td>" . $row['7days'] . "</td><td>" . $row['30days'] . "</td></tr>";
} 
} else if(isset($_GET['name'])) {
echo "<table border=\"1\" width=\"80%\">\n";
echo "<tr><th>ID</th><th>Name</th><th>Min Price</th><th>Mid Price</th><th>Max Price</th><th>7Days</th><th>30days</th></tr>\n";
$name = mysql_real_escape_string($_GET['name']);
$query = mysql_query("SELECT * FROM `items` WHERE `itemaname` LIKE '%$name%' DESC");
while($row = mysql_fetch_array($query)) {
	echo "<tr><td>" . $row['itemid'] . "</td><td>" . $row['itemname'] . "</td><td>" . $row['minprice'] . "</td><td>" . $row['midprice'] . "</td><td>" . $row['maxprice'] . "</td><td>" . $row['7days'] . "</td><td>" . $row['30days'] . "</td></tr>";
}
}
?>
<form>
Search by ID: <input type="text" name="id" /> <input type="submit" value="Search" /><br />
</form>
<form>
Search by Name(Keywords): <input type="text" name="name" /><input type="submit" value="Search" />
</form>

 

 

Can anyone find anything rong with this script is so please help

 

Thanks in advanced

 

Feel free to add on msn lassox@live.co.uk if its easyer

 

EDITED BY akitchin: removed the obnoxiously large font - please avoid using ultra-sized fonts in future posts.

[JonnoTheDev]

Having trouble reading this post!!

No wonder you got banned. You are trying to send 20000 requests to itemdb-rs.runescape.com without any pause in between requests. If that was my domain I would shoot you!

 

while($i < 20000) {
ini_set('max_execution_time', -1);
$file = @fopen("http://itemdb-rs.runescape.com/viewitem.ws?obj=" . $i, "rb");

 

[JonnoTheDev]

Also you would use CURL to do this job not fopen()!

You should set a pause between requests i.e. 20 seconds

[corbin]

With a 20 second pause, it would take 4 and a half days to complete.

 

 

Chances are if Jagex didn't make it accessible with an API or something, they don't want 3rd party sites stealing content from it.

 

 

Also, since it's based on ingame play, it will constantly be changing, meaning your data would have the potential to be constantly out of date.

 

Hate to be a kill joy, but I suggest giving up.  Why would visitors to your site use your DB when they can just use the official one?

[JonnoTheDev]

Just looked at the site being scraped. Do as corbin said.

[jonsjava]

Um...if you *must* do it this way, you could do it like this (but I agree that you shouldn't do this at all):

 

<?php
$sql_query = "";
while($i < 20000) {
ini_set('max_execution_time', -1);
$file = @fopen("http://itemdb-rs.runescape.com/viewitem.ws?obj=" . $i, "rb");
$contents = '';
while(!feof($file)) {
	$contents .= fread($file, 8192);
}
fclose($file);
$lines = explode("\n", $contents);
if($lines[207] == 'The item you were trying to view could not be found.') {
	echo $lines[207] . "<br />";
} else {
	//print_r($lines);
	$lines[225] = explode(" ", $lines[225]);
	$day = str_replace(array('class="rise">', "</span>", 'class="drop">', 'class="stay">'), "", $lines[225][3]);
	$lines[228] = explode(" ", $lines[228]);
	$days = str_replace(array('class="rise">', '</span>', 'class="drop">', 'class="stay">'), "", $lines[228][3]);
	$min = explode(" ", $lines[214]);
	$mid = explode(" ", $lines[217]);
	$max = explode(" ", $lines[220]);
	$itemname = addslashes($lines[205]);

	$sql_query .= "INSERT INTO `items` VALUES ('$i', 'http://itemdb-rs.runescape.com/viewitem.ws?obj=" . $i . "', '$itemname', '$min[2]', '$max[2]', '$mid[2]', 'NOW()', '$day', '$days');\n";
}
$i++;
}
mysql_query($sql_query) or die("There seems to be a problem: ".mysql_error());

This way, you only open up 1 MySQL connection, and send all queries through that one connection. It will go faster, as well, but it will be murder on the server.