curl with variables - nytimes

[zrosen88]

Hey all,

 

I'm kinda new here (as a poster at least) and any help would be great!

 

I'm trying to use cURL to save a news article from nytimes.com -- my script is:

 

#!/usr/local/bin/php -q
<?php
echo exec("curl -o /path/to/server/homecontent/vars.txt http://www.nytimes.com/2008/07/08/business/08fannie.html?_r=1&partner=rssnyt&emc=rss&oref=slogin");
?>

 

I know the script is fine, because when I change the nytimes link to something else (i.e. http://www.google.com/) it works exactly as it should.

 

I thought it could be due to the variables at the end, but even cutting out the

?_r=1&partner=rssnyt&emc=rss&oref=slogin

makes no difference for me.

 

Any help would be greatly appreciated!

 

Zack

[kenrbnsn]

Why aren't you using the PHP cURL functions for this?

 

Ken

[zrosen88]

I was under the impression what I am using was related to the PHP cURL functions. I'll take a look around at them, though, thanks!

 

Any idea why what I'm currently using isn't working for the NYTimes site, however?

 

Much thanks, Ken!

[zrosen88]

So I tried another idea, using more cURL functions:

<?php
$ch = curl_init("http://www.nytimes.com/2008/07/08/business/08fannie.html?_r=1&partner=rssnyt&emc=rss&oref=slogin");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
$output = curl_exec($ch);

$fh = fopen("x.txt","w");
fwrite($fh, $output);
fclose($fh);
?>

 

The file, x.txt, is created, however it is blank. All permissions are as they should be (0777).

 

Zack

[DarkWater]

How about:

 

$output = file_get_contents("http://www.nytimes.com/2008/07/08/business/08fannie.html?_r=1&partner=rssnyt&emc=rss&oref=slogin");

file_put_contents("x.txt", $output);

 

o_O

[zrosen88]

Hey DarkWater,

 

Thanks, but unfortunately my host doesn't allow that (Dreamhost). Fantastic service, fantastic support, fantastic uptime, but they don't allow file_get_contents

 

In a measure to improve security, DreamHost has disabled the PHP option allow_url_fopen. This would normally allow a programmer to open, include or otherwise use a remote file using a URL, rather than a local file path.

 

Zack

[kenrbnsn]

I tried your code and I had to add

<?php
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
?>

to get it to work.

 

Ken

[zrosen88]

Hey Ken --

 

You mean like this?

 

<?php
$ch = curl_init("http://www.nytimes.com/2008/07/08/business/08fannie.html?_r=1&partner=rssnyt&emc=rss&oref=slogin");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

$output = curl_exec($ch);

$fh = fopen("x.txt","w");
fwrite($fh, $output);
fclose($fh);
?>

 

If so, still nothing  ??? Unless I did something I shouldn't have; I'm relatively new to cURL, and apologize for any ignorance!

 

Zack

[kenrbnsn]

This is the code I used that worked (with my debugging statements still in place):

<?php
$ch = curl_init("http://www.nytimes.com/2008/07/08/business/08fannie.html?_r=1&partner=rssnyt&emc=rss&oref=slogin");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$output = curl_exec($ch);
echo '<pre>curl_getinfo:' . print_r(curl_getinfo($ch),true) . '</pre>';
if ($output == '') echo 'Error: ' . curl_error($ch). '<br>';
else {
echo '<pre>Output: ' . htmlentities($output) . '</pre>';
$fh = fopen("x.php","w");
fwrite($fh, $output);
fclose($fh);
}
?>

 

Ken

[zrosen88]

Ken --

 

Perfect! That did it! 

 

Now I just have to personally figure out how to work around the advertisements that NYTimes puts randomly between viewing articles!

 

Thanks again! Couldn't have done it without you!

 

Zack Rosen, Website Director

88.7FM WNHU

CT's #1 College Radio Station

http://www.wnhu.net/