Command Line curl and PHP curl Different results

[codeinphp]

I am attempting to curl a page to extract some data. I can pass the information thru curl from command line and get exactly what is needed.  If I create a php file with the same information and execute the php I get a 200 response but the return is empty. I can't seem to figure it out. Here is the the script.

<?php

$headers = array();
$headers[]= 'Host: www.streamlive.to';
$headers[]= 'Connection: keep-alive';
$headers[]= 'Content-Length: 27';
$headers[]= 'Pragma: no-cache';
$headers[]= 'Cache-Control: no-cache';
$headers[]= 'Accept: */*';
$headers[]= 'Origin: http://www.streamlive.to';
$headers[]= 'X-Requested-With: XMLHttpRequest';
$headers[]= 'User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36';
$headers[]= 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8';
$headers[]= 'Referer: http://www.streamlive.to/tv-guide';
$headers[]= 'Accept-Encoding: gzip, deflate';
$headers[]= 'Accept-Language: en-US,en;q=0.8';
$headers[]= 'Cookie: PHPSESSID=f7mqqpgm3beto9a462fn8a2pl3; _gat=1; _ga=GA1.2.1850053691.1432824439';

$ch = curl_init('http://www.streamlive.to/tv-ajax.php');

curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);

curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_HEADER,true);

curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

curl_setopt($ch, CURLOPT_VERBOSE, true);//0-FALSE  1 TRUE
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode('getchannelsched&id=9'));
curl_exec($ch);
$html = curl_exec($ch);

curl_close($ch);
var_dump($html);



if (file_put_contents ('data.xml', $html) !== false) {
    echo 'Success!';
} else {
     echo 'Failed';
}



?>

Here is the command line curl

curl -o test.txt "http://www.streamlive.to/tv-ajax.php" -H "Pragma: no-cache" -H "Origin: http://www.streamlive.to" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US,en;q=0.8" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" -H "Accept: */*" -H "Cache-Control: no-cache" -H "X-Requested-With: XMLHttpRequest" -H "Cookie: PHPSESSID=f7mqqpgm3beto9a462fn8a2pl3; _gat=1; _ga=GA1.2.1850053691.1432824439';" -H "Connection: keep-alive" -H "Referer: http://www.streamlive.to/tv-guide" --data "action=getchannelsched&id=9" --compressed

[jcbones]

I notice you have a Content-Length in your PHP script, but not in your command line.  Not sure if the page is cutting the data off, or waiting for more.  Are you sure the passed data is 27 octets?

14.13 Content-Length

The Content-Length entity-header field indicates the size of the entity-body, in decimal number of OCTETs, sent to the recipient or, in the case of the HEAD method, the size of the entity-body that would have been sent had the request been a GET.

Content-Length = "Content-Length" ":" 1*DIGIT

An example is

Content-Length: 3495

Applications SHOULD use this field to indicate the transfer-length of the message-body, unless this is prohibited by the rules in section 4.4.

Any Content-Length greater than or equal to zero is a valid value. Section 4.4 describes how to determine the length of a message-body if a Content-Length is not given.

Note that the meaning of this field is significantly different from the corresponding definition in MIME, where it is an optional field used within the "message/external-body" content-type. In HTTP, it SHOULD be sent whenever the message's length can be determined prior to being transferred, unless this is prohibited by the rules in section 4.4.

[codeinphp]

Thanks for looking at this.  I got the Content length from the request header.  I have tried it with out the content length and with it increased with same result. Code 200 but empty.

[bsmither]

I see two executions:

curl_exec($ch);
$html = curl_exec($ch);

I have no idea if that is appropriate.

[codeinphp]

Ah, didn't even notice, forogt  to delete the first one. Thanks. I'll delete one and see if that helps.

[codeinphp]

When executed now, with var_dump($html) I still get 200 but an empty file.

 

200string(336) "HTTP/1.1 200 OK Server: nginx/1.4.3 Date: Sun, 31 May 2015 01:54:04 GMT Content-Type: text/html Transfer-Encoding: chunked X-Powered-By: PHP/5.3.3 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Encoding: gzip ‹" Success!

[codeinphp]

If I take out the Content Length I get 200 but empty return, if I leave it in I get 0 code back and empty file.

[jcbones]

If this request relies on sessions, then perhaps the garbage collector has already gobbled that up.  Meaning the sessionID doesn't exist on the server.

[codeinphp]

If sessionID is no good then the command line curl should not work since it needs the same cookie, correct?

[jcbones]

Well, I played with it a bit, and found the problem.  

curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode('getchannelsched&id=9'));

Change to:

curl_setopt($ch, CURLOPT_POSTFIELDS, 'action=getchannelsched&id=9');

Edited May 31, 2015 by jcbones

[codeinphp]

When you execute script does it create a file with information?

[jcbones]

Yes.

This is the script I used:

 

<?php
 
$headers = array();
$headers[]= 'Host: www.streamlive.to';
$headers[]= 'Connection: keep-alive';
//$headers[]= 'Content-Length: 27';
$headers[]= 'Pragma: no-cache';
$headers[]= 'Cache-Control: no-cache';
$headers[]= 'Accept: */*';
$headers[]= 'Origin: http://www.streamlive.to';
$headers[]= 'X-Requested-With: XMLHttpRequest';
$headers[]= 'User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36';
$headers[]= 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8';
$headers[]= 'Referer: http://www.streamlive.to/tv-guide';
$headers[]= 'Accept-Encoding: gzip, deflate, sdch';
$headers[]= 'Accept-Language: en-US,en;q=0.8';
$headers[]= 'Cookie: PHPSESSID=f7mqqpgm3beto9a462fn8a2pl3; _gat=1; _ga=GA1.2.1850053691.1432824439';
 
$ch = curl_init('http://www.streamlive.to/tv-ajax.php');
 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
 
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
//curl_setopt($ch, CURLOPT_HEADER,true);
 
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
 
curl_setopt($ch, CURLOPT_VERBOSE, true);//0-FALSE  1 TRUE
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'action=getchannelsched&id=14');
 
$html = curl_exec($ch);
$html = gzdecode($html);
 
curl_close($ch);
 
echo '<pre>';
var_dump($html);
echo '</pre>';
 
if (file_put_contents ('data.txt', $html) !== false) {
    echo 'Success!';
} else {
     echo 'Failed';
}

[jcbones]

This is the result: Chromium****

<div class="row thead">
<input class="fullgrid_btn" type="button" value="View All Channels" />
Daily Programming on <span class="channel_name">BRAVO</span> for Sunday, May 31, 2015 thru Sunday, June 7, 2015 
</div>
<div class="tbody">
<div class="row">
<div class="col th">
12:00 AM <br/> May 31, 2015 </div>
<div class="prog_cols" >
<div class="col ts prog_910899 movies" data-progid="910899" data-catid="movies" >
<span class="prog_name">Enough</span>
<div class="prog_time">May 31, 2015, 12:00 am - 2:00 am</div>
<a class="btn_watchlist disabled" href="javascript:void(0)" data-progid="910899">(+) add to watchlist</a>
<div class="prog_desc">A woman on the lam with her young daughter takes drastic steps when her abusive husband tracks them down.</div>
</div>
</div>
<a class="watchnow" href="http://www.streamlive.to/premium">Watch Now</a>
</div>
<div class="row">
<div class="col th">
2:00 AM <br/> May 31, 2015 </div>
<div class="prog_cols" >
<div class="col ts prog_910900 " data-progid="910900" data-catid="" >
<span class="prog_name">The Real Housewives of Atlanta</span>
<div class="prog_time">May 31, 2015, 2:00 am - 3:00 am</div>
<a class="btn_watchlist disabled" href="javascript:void(0)" data-progid="910900">(+) add to watchlist</a>
<div class="prog_desc">Kandi Burruss and Todd Tucker plan a family ski trip to Colorado.
</div>
</div>
</div>
<a class="watchnow" href="http://www.streamlive.to/premium">Watch Now</a>
</div>
<div class="row">
<div class="col th">
3:00 AM <br/> May 31, 2015 </div>
<div class="prog_cols" >
<div class="col ts prog_910901 " data-progid="910901" data-catid="" >
<span class="prog_name">Blood, Sweat & Heels</span>
<div class="prog_time">May 31, 2015, 3:00 am - 4:00 am</div>
<a class="btn_watchlist disabled" href="javascript:void(0)" data-progid="910901">(+) add to watchlist</a>
<div class="prog_desc">Chantelle has a fast-paced blind date with a sightly bachelor while Mica has an intimate date at her apartment. In other developments, Demetria's cake-tasting gathering takes a sour turn; the crew heads to the Hamptons; and a painful moment tests Daisy.</div>
</div>
</div>
<a class="watchnow" href="http://www.streamlive.to/premium">Watch Now</a>
</div>
<div class="row">
<div class="col th">
4:00 AM <br/> May 31, 2015 </div>
<div class="prog_cols" >
<div class="col ts prog_910902 " data-progid="910902" data-catid="" >
<span class="prog_name">Paid Programming</span>
<div class="prog_time">May 31, 2015, 4:00 am - 6:00 am</div>
<a class="btn_watchlist disabled" href="javascript:void(0)" data-progid="910902">(+) add to watchlist</a>
<div class="prog_desc">No details are there for this program.</div>
</div>
</div>
<a class="watchnow" href="http://www.streamlive.to/premium">Watch Now</a>
</div>
<div class="row">
<div class="col th">
6:00 AM <br/> May 31, 2015 </div>
<div class="prog_cols" >
<div class="col ts prog_910903 " data-progid="910903" data-catid="" >
<span class="prog_name">Million Dollar Listing New York</span>
<div class="prog_time">May 31, 2015, 6:00 am - 8:00 am</div>
<a class="btn_watchlist disabled" href="javascript:void(0)" data-progid="910903">(+) add to watchlist</a>
<div class="prog_desc">Fredrik's reputation is tested when developers threaten to pull out of premium finishes they promised. Elsewhere, Ryan reunites with a familiar blonde, and Luis pursues one of the city's top developers.</div>
</div>
</div>
<a class="watchnow" href="http://www.streamlive.to/premium">Watch Now</a>
</div>
<div class="row">
***Truncated for posting***

[codeinphp]

Thanks again. I will go thru your code and find my error. When I run your script I get a proper var_dump.

[codeinphp]

What does your data.txt file look like? My result to browser is correct but nothing is saved to file.

[codeinphp]

Never mind, my mistake.  I am getting info back.  Once again thank you for taking your time to help me.