Web Crawling Experiments - A Variety Ways of Coding

[phpsane]

Hi Folks,

 

In this thread, we experiment with a variety of ways to crawl the web with php.

I will start this thread off by showing some of the codes I am working on.

Edited July 29, 2018 by phpsane

[phpsane]

Php Lovers,

 

Have a look at the following code. It works to fetch google.com but fails to fetch a serp from fiver.com. Any ideas why this is the case ?

Let's try to find the problem first and then the solution. I for one, have failed to find the reason why the fiverr.com serp page fetching is failing.

CODE 1

	<?php 
	# Use the Curl extension to query Google and get back a page of results
$url = "http://google.com"; //Working with this url.
//Why not working with following url ?
//$url = "https://www.fiverr.com/search/gigs?utf8=%E2%9C%93&source=guest-homepage&locale=en&search_in=everywhere&query=php";
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$html = curl_exec($ch);
curl_close($ch);
	# Create a DOM parser object
$dom = new DOMDocument();
	# Parse the HTML from Google.
# The @ before the method call suppresses any warnings that
# loadHTML might throw because of invalid HTML in the page.
@$dom->loadHTML($html);
	# Iterate over all the <a> tags
foreach($dom->getElementsByTagName('a') as $link) {
        # Show the <a href>
        echo $link->getAttribute('href');
        echo "<br />";
        echo $link->nodeValue;    
        echo "<br />";
}
	?>
	

[requinix]

No.

This was one of the bad things you did before that got you kicked out. Don't do it again.