cURL Problem

[A1Dan]

Hi,

 

I've been using a cURL script to gather stats from a site for a while, and it appears to have stopped working.  The script now just hangs, eventually timing out.  The problem only affects one particular domain from which I am attempting to gather data, and can be demonstrated with the following test script I have set up (source below), to just copy a file given in the ?URL parameter into an array and echo the contents:

 

http://www.aidangrant.com/curltest.php?URL=www.bbc.co.uk

 

As you should see, the above works fine, but if I try the problematic URL, it does not:

 

http://www.aidangrant.com/curltest.php?URL=www.fantasyleague.com

 

Any idea what would be causing this?  As I say, this was working fine until recently...

 

Cheers

 

Aidan

 

 

Source of curltest.php:

 

<?php

  $ch = curl_init();

  $timeout = 180; // set to zero for no timeout 

  curl_setopt ($ch, CURLOPT_URL, $_GET['URL']);

  curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

  curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);

  $file_contents = curl_exec($ch);

  curl_close($ch);

  $lines = array();

  $lines = explode("\n", $file_contents);

  if (!isset($lines)){

    $lines=array('');

  }

  foreach ($lines as $line_num => $line) {   

    echo $line;

  }

?>

[trq]

Could be anything but its likely the server is blocking requests from curl. have you got permissions to whatever it is you end up doing with your curl requests?

[A1Dan]

Could be anything but its likely the server is blocking requests from curl. have you got permissions to whatever it is you end up doing with your curl requests?

 

Not sure I follow (sorry probably being thick!), do you mean read permissions on the 3rd party server?  Does everybody not have read permissions, to be able to view the web-page?

[A1Dan]

Hi Thorpe - could you elaborate a bit on what I need to check? 

 

Thanks

 

Aidan

[trq]

There isn't anything you need to check, however, its not uncommon for server admins to block requests from clients such as curl.

 

You could try setting the user agent to something to make your request look lore like a normal browser.

 

However, having said that, I just ran your script from one of my servers without issue.

[A1Dan]

Cheers Thorpe - I've just tried running the script from my home machine and it works - does this mean it is likely my host's web-server may be blocked? (on its IP?).

How sould this happen, and is there anyway to confirm it is the case?

[DarkerAngel]

The only thing I could suggest is maybe a ping command if you have SSH access, Either that or I have this:

 

/**
* Quick cURL Function
*
* @param string $url URL To Site access
* @param array $postdata Data Array to Send to Page in POST Form
* @param string $refer Referring URL
*
* @return string HTML Document, Text, or Content Data returned from the site.
*/
function gethtml($url, $postdata = false, $refer = false) {
	$ch = curl_init(); 
		curl_setopt($ch, CURLOPT_URL,$url );
		curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
		if($postdata) {
			curl_setopt($ch, CURLOPT_POST, 1);
			curl_setopt($ch, CURLOPT_POSTFIELDS,$postdata);
		}
		curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5");
		curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
		curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
		curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
		curl_setopt($ch, CURLOPT_REFERER, $refer);
		curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
	$html = curl_exec ($ch);
    curl_close($ch);
	return $html;
}

 

/*** Example of how to call the function ***/
$content = gethtml("http://website.com/");

/*** Example of Posting Fields ***/

$post = array();
$post['field_name'] = "field value";

$content = gethtml("http://website.com/form.php", $post);

/**
* The Last parameter is for sites that refuse to return content with out a referral header
*/

[makkusuo]

First i have to say i'm amazed this is already indexed in google. it's right up there with my issue.

 

I have a script using curl to process transactions on one of my sites. it's been this way for a year and now it stopped working 2 days ago. i've done a test in ssh as follows:

 

time curl -s https://www.domainname.com (dummy domain, put whatever you want in its place)

 

what i get is nothing... a blank response. this is from my server (a Plesk VPS). However when i try this SSH test on another server (outside my network) i get the html output im supposed to see.

 

Also, i can get an html response from any website within my server (the Plesk VPS), just not outside websites. I used to be able to get them no problem. this is causing me to not get orders since i'm getting blank responses.

 

Any thoughts on what might be the problem?

[A1Dan]

Thanks DarkerAngel - same outcome with your script though, unfortunately.  I don't seem to be able to ping from the ssh shell - it's very limited jailshell.

[trq]

Thanks DarkerAngel - same outcome with your script though, unfortunately.  I don't seem to be able to ping from the ssh shell - it's very limited jailshell.

 

Can you actually ping this server from your vps? eg;

 

ping -c 1 www.fantasyleague.com

[A1Dan]

Can you actually ping this server from your vps? eg;

 

ping -c 1 www.fantasyleague.com

 

Nope - I get "ping: icmp open socket: Operation not permitted", which I get if I try and ping any URL from it, so presume is down to limitations on what I have permission to do in the shell?

 

Oddly, even if I ping www.fantasyleague.com from my own pc it times out 4 out of 4, which seems weird....