Failed to open stream: HTTP Request failed.

[sogorman]

I am trying to figure how to code around this. I have a DOM scraper function that pulls urls from my database, opens the page, scrapes the data, and then moves onto the next URL to scrape. my issue is that if the page fails to load the script bombs and I have to restart it again. Trying to figure out how if the

$html->find('div[class="itemHeader address"]') as $div

fails to open the page it just skips the DOM inspection.

 

Here is my error...

 

Failed to open stream: HTTP Request failed.

 

Here is where I am at with my script....

 

mysql_select_db("scraped") or die(mysql_error());
$result = mysql_query("SELECT PKEY, URL, HASSCRAPED, SHOULDSCRAPE FROM CRAWLED WHERE SHOULDSCRAPE ='1' AND HASSCRAPED = '0'") or die(mysql_error());  

while($row = mysql_fetch_array( $result )) {
	mysql_query("UPDATE CRAWLED SET CRAWLED.HASSCRAPED =  '1' WHERE CRAWLED.URL = '" . $row['URL'] . "'");
	$html = file_get_html($row['URL']);
	foreach($html->find('div[class="itemHeader address"]') as $div) {
		foreach($div->find('a') as $element){
			$CleanData = CleanData($element->innertext);
			if (strlen($CleanData[0]) > 5){
   				mysql_query("INSERT INTO SCRAPED (ADDR1, CITY, STATE, ZIP, URL, DATE) VALUES ('" . $CleanData[0] . "','" . $CleanData[1] . "','" . $CleanData[2] . "','" . $CleanData[3] . "','". $row['URL'] . "','". date( 'Y-m-d H:i:s ' ) ."')");
   	   			}
		}
	}

	$html->clear(); 
	unset($html);
	unset($CleanData);
} 

function CleanData($data) {
  $NewData = trim($data);
  $NewData = str_replace("<em>", "", $NewData);
  $AddrCityStateZip = explode("</em>",$NewData);
  $CityStateZip = explode(",",$AddrCityStateZip[1]);
  $StateZip = explode(' ',$CityStateZip[1]);
  $NewDataArray = array ($AddrCityStateZip[0], $CityStateZip[0], $StateZip[1], $StateZip[2]);
  return $NewDataArray;
  unset($NewData);
  unset($AddrCityStateZip);
  unset($CityStateZip);
  unset($StateZip);
}
mysql_close($link);
echo 'Scraping has compleated without error';

[gizmola]

My suggestion is to use curl