rofl90 Posted June 11, 2008 Share Posted June 11, 2008 Hi guys I have a list of sites that have been generated by a script, eg www.someweb.com/a-a-a-d102289.html how could I crawl each of them, take some information from a specific div for instance ATTRACTION_REVIEW and make a query to insert the contents of the div into a database Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 Is there a way to ONLY get that div without looking through it? Quote Link to comment Share on other sites More sharing options...
DarkWater Posted June 11, 2008 Share Posted June 11, 2008 You could use regular expressions... Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 Could you giv eme a code example to get it from a site? Quote Link to comment Share on other sites More sharing options...
DarkWater Posted June 11, 2008 Share Posted June 11, 2008 Give me an exact example of a div that you'd want extracted. Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 <div id="ATTRACTION_REVIEW" class="listing"> <div class="details"> <div class="information textual"> <div class="description"> <b>Traveler Description:</b> <div class="toggle"> <span class="onHide">Attractions include favorites such as the Log Flume, Dodgems, Adventure Golf, Pan for Gold or brave the Horror Hotel. Located next to the sandy... <span class="show">more »</span> </span> <span class="onShow">Attractions include favorites such as the Log Flume, Dodgems, Adventure Golf, Pan for Gold or brave the Horror Hotel. Located next to the sandy beaches, marina and working harbour at Littlehampton in West Sussex. It has all you need for a great family day out. New for 2008 - Indoor Cannon Ball Play Zone & Caterpillar Roller Coaster. Book online for family and group discounts. <span class="hide">« less</span> </span> </div> </div> <div class="description addDesc"> Familiar with Harbour Park? Write your own description and <a href="/Travel-g504226-d1005323/Littlehampton:United-Kingdom:Harbour.Park.html" rel="nofollow">share what you know</a> with other travelers.<br/> </div> </div><!--/ information.textual--> <div class="information textual"> <div class="type"><b>Attraction type:</b> Amusement/theme park</div> </div><!--/ information.textual--> <div class="information bulleted"> <ul class="arrows"> <li><a target="_blank" href="http://www.harbourpark.com/">http://www.harbourpark.com/</a></li> <li><a href="mailto:fun@harbourpark.com">fun@harbourpark.com</a></li> </ul> </div><!--/ information.bulleted--> <div class="information textual adr"> <div> <b>Address:</b> <span class="street-address">Seafront</span> <span class="locality"> Littlehampton BN17 5LL</span> <br/><span class="country-name">England</span> </div> <div><b>Tel:</b> <span class="tel">01903 721200</span></div> <div><b>Fax:</b> <span class="fax">01903 716663</span></div> </div><!--/ information.textual--> </div><!--/ details--> </div><!--/ ATTRACTION_REVIEW.listing--> Quote Link to comment Share on other sites More sharing options...
DarkWater Posted June 11, 2008 Share Posted June 11, 2008 The WHOLE contents? Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 Yeh Quote Link to comment Share on other sites More sharing options...
DarkWater Posted June 11, 2008 Share Posted June 11, 2008 You're luck that comment is there (the one that says that ATTRACTION_REVIEW.listing is over), otherwise you'd be screwed. Here: $file = file_get_contents("http://something.com/something.html"); $match = array(); preg_match('~<div id="ATTRACTION_REVIEW" class="listing">(.+?)</div><!--/ ATTRACTION_REVIEW.listing-->~i', $file, $match); Then, $match[1] will have the content in that DIV. Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 Thanks - this is my current code - <?php ob_start(); $CONF = array(); $CONF["DATABASE"] = 'x'; $CONF["USERNAME"] = 'x'; $CONF["PASSWORD"] = 'x'; $CONF["HOST"] = 'x'; mysql_connect($CONF["HOST"], $CONF["USERNAME"], $CONF["PASSWORD"]); mysql_select_db($CONF["DATABASE"]); $first = 100000; while($first < 100001) { $fileName = "http://www.x.com/x-g--d" . $first . ".html"; $file = file_get_contents($fileName); $match = array(); if(preg_match('~<div id="ATTRACTION_REVIEW" class="listing">(.+?)</div><!--/ ATTRACTION_REVIEW.listing-->~i', $file, $match)) { if(mysql_query("INSERT INTO pages (linkName, theText) VALUES('$fileName', '$match[1]')")) { echo "Success\n\n"; $first++; } else { echo "Failure\n\n"; } } } ob_end_flush(); ?> It just continuously loads. Quote Link to comment Share on other sites More sharing options...
DarkWater Posted June 11, 2008 Share Posted June 11, 2008 $first must not be getting incremented because this is failing: mysql_query("INSERT INTO pages (linkName, theText) VALUES('$fileName', '$match[1]')") Instead of using it in the if statement, do this: $result = mysql_query("INSERT INTO pages (linkName, theText) VALUES('$fileName', '$match[1]')") OR die(mysql_error()); //important if ($result) { } Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 Unfortunately still a continual load. Heres code, minus the mysql stuff: mysql_connect($CONF["HOST"], $CONF["USERNAME"], $CONF["PASSWORD"]); mysql_select_db($CONF["DATABASE"]); $first = 100000; while($first < 100001) { $fileName = "http://www.x.com/x-g--d" . $first . ".html"; $file = file_get_contents($fileName); $match = array(); if(preg_match('~<div id="ATTRACTION_REVIEW" class="listing">(.+?)</div><!--/ ATTRACTION_REVIEW.listing-->~i', $file, $match)) { $result = mysql_query("INSERT INTO pages (linkName, theText) VALUES('$fileName', '$match[1]')") or die(mysql_error()); //important if($result) { echo "Success\n\n"; $first++; } else { echo "Failure\n\n"; } } } ob_end_flush(); ?> Quote Link to comment Share on other sites More sharing options...
DarkWater Posted June 11, 2008 Share Posted June 11, 2008 Are you sure it's retrieving the file? Also, add an else to the preg_match part that breaks that loop and echos an error. Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 Still continually loading: code: $first = 100000; while($first < 100001) { $fileName = "http://www.x.com/x-g--d" . $first . ".html"; $file = file_get_contents($fileName); $match = array(); if(preg_match('~<div id="ATTRACTION_REVIEW" class="listing">(.+?)</div><!--/ ATTRACTION_REVIEW.listing-->~i', $file, $match)) { $result = mysql_query("INSERT INTO pages (linkName, theText) VALUES('$fileName', '$match[1]')") or die(mysql_error()); //important if($result) { echo "Success\n\n"; $first++; } else { echo "Failure\n\n"; } } else { echo "Error: preg_match part did not work"; } } ob_end_flush(); ?> Quote Link to comment Share on other sites More sharing options...
DarkWater Posted June 11, 2008 Share Posted June 11, 2008 Make it: echo "Error: preg_match failed."; break; Instead of what you have. Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 get rid of the braces? Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 with the braces it gave me: Error: preg_match part did not work Quote Link to comment Share on other sites More sharing options...
DarkWater Posted June 11, 2008 Share Posted June 11, 2008 Yeah, you need the braces, lol. Try changing the regex to: ~<div id="ATTRACTION_REVIEW" class="listing">(.+?)</div><!--/ ATTRACTION_REVIEW\.listing-->~i Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 same error Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 does the . in ATTRACTION_REVIEW\.listing need to be escaped? Quote Link to comment Share on other sites More sharing options...
DarkWater Posted June 11, 2008 Share Posted June 11, 2008 That's why I put that \, lol. Try removing the ? after the + sign in the ( ). Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 still giving off the error Quote Link to comment Share on other sites More sharing options...
DarkWater Posted June 11, 2008 Share Posted June 11, 2008 Hmm...Make sure that $file is being filled correctly. Quote Link to comment Share on other sites More sharing options...
rofl90 Posted June 11, 2008 Author Share Posted June 11, 2008 Just tested it, it definitely is, it MUST be a problem with the regex, is it because it has a new line maybe? (\n)? Quote Link to comment Share on other sites More sharing options...
DarkWater Posted June 11, 2008 Share Posted June 11, 2008 Yes, forgot that . doesn't cover new lines. Try adding \n to the regex inside of the ( ). ([.\n]+) Make it that. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.