Jump to content

Problem in using the CURL to get html source code of a url


ammu412

Recommended Posts

Hi friends,

 

<?php

ini_set("display_errors",1);

error_reporting(E_ALL);

$url = "http://www.yahoo.com";

$curl = curl_init($url);

echo $curl;

curl_setopt($curl, CURLOPT_HEADER, FALSE);

curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);

curl_setopt($curl, CURLOPT_FOLLOWLOCATION, TRUE);

curl_setopt($curl, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);

$source = curl_exec($curl);

curl_close ($curl);

echo $source;

?>

 

In the above code, i didn't get anything as output.

 

In php ini:

CURL support enabled

CURL Information libcurl/7.15.1 OpenSSL/0.9.8a zlib/1.2.3 libidn/0.6.2

 

Help me to solve this problem.............

 

 

 

Link to comment
Share on other sites

$ch = curl_init();

curl_setopt($ch, CURLOPT_USERAGENT, 'WhateverBrowser1.45');

curl_setopt($ch, CURLOPT_URL, 'http://www.yahoo.com');

curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);

curl_setopt ($ch, CURLOPT_TIMEOUT, 60);

$result = curl_exec($ch);

echo "<pre>$result</pre>";

Link to comment
Share on other sites

Hi sKunKbad,

 

    Can i give $_SERVER['HTTP_USER_AGENT'] instead of WhateverBrowser1.45.

 

What do you mean? You'll already using $_SERVER['HTTP_USER_AGENT'] aren't you?

 

You may also want to check for errors:

 

if(curl_errorno($ch)){
    echo 'Error: '.curl_error($ch);
}

 

 

Link to comment
Share on other sites

Hi all,

 

I have had endless problems with CURL before. I found it not to be reliable, ie it did not work 100% of the time.

 

Have a look at this way of getting the HTML source:

 

I use Simon Willsons PHP HTTPClient class, like this:

 

require_once("./includes/classes/class_http_client.php");

$client = new HttpClient('google.com');

if (!$client->get('/')) {
  die('An error occurred: '.$client->getError());
}

$html = $client->getContent();
//the homepage source for google.com will be in the $html variable

 

How easy was that? You can even do POST with it.

 

Want to fake the user agent string? That's easy too...

 

$client->setUserAgent('CoolBrowser 1.0');

 

Make sure you include the class, get the class from http://scripts.incutio.com/httpclient/

Link to comment
Share on other sites

GingerRobot,

 

I probably was not using it properly, but I did not have time to research what all the curl flags mean and debug it.

 

It was working 98% of the time, the other 2% the post data was not being sent, which meant I had to manually post those.

 

IMO the fact that this thread exists, and my experience with it means that CURL is not easy to use.

 

The code I posted uses a raw TCP socket wrapped in the HTTP protocol (1.1 and 1.0 available), I can see exactly what this code is doing.

 

There are loads of options, like as mentioned setting the user agent, using get, post, http 1.0 or 1.1. I am currently using it to put an adsl checker on a site I am working on.

Link to comment
Share on other sites

An intermittent fault is more than likely caused by a fault at the receiving site (something which can happen regardless of your method of making the request) and is something that should be checked for and dealt with (again, regardless of your method).

 

IMO the fact that this thread exists, and my experience with it means that CURL is not easy to use.

 

By that logic, everything in PHP would be hard to use. Think of a potential problem and there'll be a thread on it.

 

There are loads of options, like as mentioned setting the user agent, using get, post, http 1.0 or 1.1.

 

Like cURL you mean?

 

Don't get me wrong, i'm not saying you have to use cURL - i'm just stating that there are many methods of achieving the same thing. When it comes down to it, there's probably not a lot of difference. However, you can't just say that, because you had problems, x method is rubbish.

Link to comment
Share on other sites

By that logic, everything in PHP would be hard to use. Think of a potential problem and there'll be a thread on it.

 

I am not saying that because a thread exists it means it's difficult to use. I am saying because of my experience and along with all the threads I see here about curl (must be 3 or 4 within a morning).

 

Surely you can't deny that curl overly complicated easy to use.

 

Like cURL you mean?

 

curl probably can do all of these things, but I don't want to have to learn all the curl flags and the combination of which should be used for a certain task, when there is an alternative which is much easier.

 

Don't get me wrong, i'm not saying you have to use cURL - i'm just stating that there are many methods of achieving the same thing. When it comes down to it, there's probably not a lot of difference. However, you can't just say that, because you had problems, x method is rubbish.

 

Agreed there are many ways of doing something. However there is a lot of difference in the way the two methods in question are used.

 

Aside from that curl is written in a procedural fashion. I personally prefer to program in an OOP fashion. Curl is very old and has not been updated inline with todays programming. Yes you could wrap it up in a class but I wouldn't want to be then one to have to do that. :)

 

 

Link to comment
Share on other sites

By that logic, everything in PHP would be hard to use. Think of a potential problem and there'll be a thread on it.

 

I am not saying that because a thread exists it means it's difficult to use. I am saying because of my experience and along with all the threads I see here about curl (must be 3 or 4 within a morning).

 

Surely you can't deny that curl overly complicated easy to use.

 

Like cURL you mean?

 

curl probably can do all of these things, but I don't want to have to learn all the curl flags and the combination of which should be used for a certain task, when there is an alternative which is much easier.

 

Don't get me wrong, i'm not saying you have to use cURL - i'm just stating that there are many methods of achieving the same thing. When it comes down to it, there's probably not a lot of difference. However, you can't just say that, because you had problems, x method is rubbish.

 

Agreed there are many ways of doing something. However there is a lot of difference in the way the two methods in question are used.

 

Aside from that curl is written in a procedural fashion. I personally prefer to program in an OOP fashion. Curl is very old and has not been updated inline with todays programming. Yes you could wrap it up in a class but I wouldn't want to be then one to have to do that. :)

 

 

 

I consider cURL easy to use, and an essential part of my php apps. I've used it in many scripts, and my experience is that it is reliable. It does require more than just copy and pasting of code; it requires knowledge; and there are many tutorials out there that will get you started. We can't hold your hand every step of the way, and much of programming and php can't be figured out in a brief skim of a webpage. You'll have to read, and read, and read some more.

 

What I think is funny, is that I provided a reliable solution to the original post, and this thread turned into an argument on how easy/hard cURL is to use. If it were so hard to use, then my code wouldn't work, but it does. Try it.

Link to comment
Share on other sites

This thread is more than a year old. Please don't revive it unless you have something important to add.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.