Problem in using the CURL to get html source code of a url

[ammu412]

Hi friends,

 

<?php

ini_set("display_errors",1);

error_reporting(E_ALL);

$url = "http://www.yahoo.com";

$curl = curl_init($url);

echo $curl;

curl_setopt($curl, CURLOPT_HEADER, FALSE);

curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);

curl_setopt($curl, CURLOPT_FOLLOWLOCATION, TRUE);

curl_setopt($curl, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);

$source = curl_exec($curl);

curl_close ($curl);

echo $source;

?>

 

In the above code, i didn't get anything as output.

 

In php ini:

CURL support enabled

CURL Information libcurl/7.15.1 OpenSSL/0.9.8a zlib/1.2.3 libidn/0.6.2

 

Help me to solve this problem.............

 

 

 

[sKunKbad]

$ch = curl_init();

curl_setopt($ch, CURLOPT_USERAGENT, 'WhateverBrowser1.45');

curl_setopt($ch, CURLOPT_URL, 'http://www.yahoo.com');

curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);

curl_setopt ($ch, CURLOPT_TIMEOUT, 60);

$result = curl_exec($ch);

echo "<pre>$result</pre>";

[ammu412]

Hi sKunKbad,

 

    Can i give $_SERVER['HTTP_USER_AGENT'] instead of WhateverBrowser1.45.

[GingerRobot]

Hi sKunKbad,

 

    Can i give $_SERVER['HTTP_USER_AGENT'] instead of WhateverBrowser1.45.

 

What do you mean? You'll already using $_SERVER['HTTP_USER_AGENT'] aren't you?

 

You may also want to check for errors:

 

if(curl_errorno($ch)){
    echo 'Error: '.curl_error($ch);
}

 

 

[the182guy]

Hi all,

 

I have had endless problems with CURL before. I found it not to be reliable, ie it did not work 100% of the time.

 

Have a look at this way of getting the HTML source:

 

I use Simon Willsons PHP HTTPClient class, like this:

 

require_once("./includes/classes/class_http_client.php");

$client = new HttpClient('google.com');

if (!$client->get('/')) {
  die('An error occurred: '.$client->getError());
}

$html = $client->getContent();
//the homepage source for google.com will be in the $html variable

 

How easy was that? You can even do POST with it.

 

Want to fake the user agent string? That's easy too...

 

$client->setUserAgent('CoolBrowser 1.0');

 

Make sure you include the class, get the class from http://scripts.incutio.com/httpclient/

[GingerRobot]

I have had endless problems with CURL before. I found it not to be reliable, ie it did not work 100% of the time.

 

I'm not being rude, but obviously you just weren't using it properly.

[the182guy]

GingerRobot,

 

I probably was not using it properly, but I did not have time to research what all the curl flags mean and debug it.

 

It was working 98% of the time, the other 2% the post data was not being sent, which meant I had to manually post those.

 

IMO the fact that this thread exists, and my experience with it means that CURL is not easy to use.

 

The code I posted uses a raw TCP socket wrapped in the HTTP protocol (1.1 and 1.0 available), I can see exactly what this code is doing.

 

There are loads of options, like as mentioned setting the user agent, using get, post, http 1.0 or 1.1. I am currently using it to put an adsl checker on a site I am working on.

[GingerRobot]

An intermittent fault is more than likely caused by a fault at the receiving site (something which can happen regardless of your method of making the request) and is something that should be checked for and dealt with (again, regardless of your method).

 

IMO the fact that this thread exists, and my experience with it means that CURL is not easy to use.

 

By that logic, everything in PHP would be hard to use. Think of a potential problem and there'll be a thread on it.

 

There are loads of options, like as mentioned setting the user agent, using get, post, http 1.0 or 1.1.

 

Like cURL you mean?

 

Don't get me wrong, i'm not saying you have to use cURL - i'm just stating that there are many methods of achieving the same thing. When it comes down to it, there's probably not a lot of difference. However, you can't just say that, because you had problems, x method is rubbish.

[the182guy]

By that logic, everything in PHP would be hard to use. Think of a potential problem and there'll be a thread on it.

 

I am not saying that because a thread exists it means it's difficult to use. I am saying because of my experience and along with all the threads I see here about curl (must be 3 or 4 within a morning).

 

Surely you can't deny that curl overly complicated easy to use.

 

Like cURL you mean?

 

curl probably can do all of these things, but I don't want to have to learn all the curl flags and the combination of which should be used for a certain task, when there is an alternative which is much easier.

 

Don't get me wrong, i'm not saying you have to use cURL - i'm just stating that there are many methods of achieving the same thing. When it comes down to it, there's probably not a lot of difference. However, you can't just say that, because you had problems, x method is rubbish.

 

Agreed there are many ways of doing something. However there is a lot of difference in the way the two methods in question are used.

 

Aside from that curl is written in a procedural fashion. I personally prefer to program in an OOP fashion. Curl is very old and has not been updated inline with todays programming. Yes you could wrap it up in a class but I wouldn't want to be then one to have to do that.

 

 

[sKunKbad]

By that logic, everything in PHP would be hard to use. Think of a potential problem and there'll be a thread on it.

 

I am not saying that because a thread exists it means it's difficult to use. I am saying because of my experience and along with all the threads I see here about curl (must be 3 or 4 within a morning).

 

Surely you can't deny that curl overly complicated easy to use.

 

Like cURL you mean?

 

curl probably can do all of these things, but I don't want to have to learn all the curl flags and the combination of which should be used for a certain task, when there is an alternative which is much easier.

 

Don't get me wrong, i'm not saying you have to use cURL - i'm just stating that there are many methods of achieving the same thing. When it comes down to it, there's probably not a lot of difference. However, you can't just say that, because you had problems, x method is rubbish.

 

Agreed there are many ways of doing something. However there is a lot of difference in the way the two methods in question are used.

 

Aside from that curl is written in a procedural fashion. I personally prefer to program in an OOP fashion. Curl is very old and has not been updated inline with todays programming. Yes you could wrap it up in a class but I wouldn't want to be then one to have to do that.

 

 

 

I consider cURL easy to use, and an essential part of my php apps. I've used it in many scripts, and my experience is that it is reliable. It does require more than just copy and pasting of code; it requires knowledge; and there are many tutorials out there that will get you started. We can't hold your hand every step of the way, and much of programming and php can't be figured out in a brief skim of a webpage. You'll have to read, and read, and read some more.

 

What I think is funny, is that I provided a reliable solution to the original post, and this thread turned into an argument on how easy/hard cURL is to use. If it were so hard to use, then my code wouldn't work, but it does. Try it.