codeinphp Posted May 31, 2015 Share Posted May 31, 2015 I am attempting to curl a page to extract some data. I can pass the information thru curl from command line and get exactly what is needed. If I create a php file with the same information and execute the php I get a 200 response but the return is empty. I can't seem to figure it out. Here is the the script. <?php $headers = array(); $headers[]= 'Host: www.streamlive.to'; $headers[]= 'Connection: keep-alive'; $headers[]= 'Content-Length: 27'; $headers[]= 'Pragma: no-cache'; $headers[]= 'Cache-Control: no-cache'; $headers[]= 'Accept: */*'; $headers[]= 'Origin: http://www.streamlive.to'; $headers[]= 'X-Requested-With: XMLHttpRequest'; $headers[]= 'User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36'; $headers[]= 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8'; $headers[]= 'Referer: http://www.streamlive.to/tv-guide'; $headers[]= 'Accept-Encoding: gzip, deflate'; $headers[]= 'Accept-Language: en-US,en;q=0.8'; $headers[]= 'Cookie: PHPSESSID=f7mqqpgm3beto9a462fn8a2pl3; _gat=1; _ga=GA1.2.1850053691.1432824439'; $ch = curl_init('http://www.streamlive.to/tv-ajax.php'); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_COOKIESESSION, true); curl_setopt($ch, CURLOPT_FRESH_CONNECT, true); curl_setopt($ch, CURLOPT_HEADER,true); curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); curl_setopt($ch, CURLOPT_VERBOSE, true);//0-FALSE 1 TRUE curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode('getchannelsched&id=9')); curl_exec($ch); $html = curl_exec($ch); curl_close($ch); var_dump($html); if (file_put_contents ('data.xml', $html) !== false) { echo 'Success!'; } else { echo 'Failed'; } ?> Here is the command line curl curl -o test.txt "http://www.streamlive.to/tv-ajax.php" -H "Pragma: no-cache" -H "Origin: http://www.streamlive.to" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US,en;q=0.8" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" -H "Accept: */*" -H "Cache-Control: no-cache" -H "X-Requested-With: XMLHttpRequest" -H "Cookie: PHPSESSID=f7mqqpgm3beto9a462fn8a2pl3; _gat=1; _ga=GA1.2.1850053691.1432824439';" -H "Connection: keep-alive" -H "Referer: http://www.streamlive.to/tv-guide" --data "action=getchannelsched&id=9" --compressed Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/ Share on other sites More sharing options...
jcbones Posted May 31, 2015 Share Posted May 31, 2015 I notice you have a Content-Length in your PHP script, but not in your command line. Not sure if the page is cutting the data off, or waiting for more. Are you sure the passed data is 27 octets? 14.13 Content-LengthThe Content-Length entity-header field indicates the size of the entity-body, in decimal number of OCTETs, sent to the recipient or, in the case of the HEAD method, the size of the entity-body that would have been sent had the request been a GET. Content-Length = "Content-Length" ":" 1*DIGIT An example is Content-Length: 3495 Applications SHOULD use this field to indicate the transfer-length of the message-body, unless this is prohibited by the rules in section 4.4. Any Content-Length greater than or equal to zero is a valid value. Section 4.4 describes how to determine the length of a message-body if a Content-Length is not given. Note that the meaning of this field is significantly different from the corresponding definition in MIME, where it is an optional field used within the "message/external-body" content-type. In HTTP, it SHOULD be sent whenever the message's length can be determined prior to being transferred, unless this is prohibited by the rules in section 4.4. Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512874 Share on other sites More sharing options...
codeinphp Posted May 31, 2015 Author Share Posted May 31, 2015 Thanks for looking at this. I got the Content length from the request header. I have tried it with out the content length and with it increased with same result. Code 200 but empty. Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512875 Share on other sites More sharing options...
bsmither Posted May 31, 2015 Share Posted May 31, 2015 I see two executions: curl_exec($ch); $html = curl_exec($ch); I have no idea if that is appropriate. Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512876 Share on other sites More sharing options...
codeinphp Posted May 31, 2015 Author Share Posted May 31, 2015 Ah, didn't even notice, forogt to delete the first one. Thanks. I'll delete one and see if that helps. Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512877 Share on other sites More sharing options...
codeinphp Posted May 31, 2015 Author Share Posted May 31, 2015 When executed now, with var_dump($html) I still get 200 but an empty file. 200string(336) "HTTP/1.1 200 OK Server: nginx/1.4.3 Date: Sun, 31 May 2015 01:54:04 GMT Content-Type: text/html Transfer-Encoding: chunked X-Powered-By: PHP/5.3.3 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Encoding: gzip ‹" Success! Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512878 Share on other sites More sharing options...
codeinphp Posted May 31, 2015 Author Share Posted May 31, 2015 If I take out the Content Length I get 200 but empty return, if I leave it in I get 0 code back and empty file. Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512879 Share on other sites More sharing options...
jcbones Posted May 31, 2015 Share Posted May 31, 2015 If this request relies on sessions, then perhaps the garbage collector has already gobbled that up. Meaning the sessionID doesn't exist on the server. Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512881 Share on other sites More sharing options...
codeinphp Posted May 31, 2015 Author Share Posted May 31, 2015 If sessionID is no good then the command line curl should not work since it needs the same cookie, correct? 1 Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512886 Share on other sites More sharing options...
jcbones Posted May 31, 2015 Share Posted May 31, 2015 (edited) Well, I played with it a bit, and found the problem. curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode('getchannelsched&id=9')); Change to: curl_setopt($ch, CURLOPT_POSTFIELDS, 'action=getchannelsched&id=9'); Edited May 31, 2015 by jcbones Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512888 Share on other sites More sharing options...
codeinphp Posted May 31, 2015 Author Share Posted May 31, 2015 When you execute script does it create a file with information? Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512889 Share on other sites More sharing options...
jcbones Posted May 31, 2015 Share Posted May 31, 2015 Yes. This is the script I used: <?php $headers = array(); $headers[]= 'Host: www.streamlive.to'; $headers[]= 'Connection: keep-alive'; //$headers[]= 'Content-Length: 27'; $headers[]= 'Pragma: no-cache'; $headers[]= 'Cache-Control: no-cache'; $headers[]= 'Accept: */*'; $headers[]= 'Origin: http://www.streamlive.to'; $headers[]= 'X-Requested-With: XMLHttpRequest'; $headers[]= 'User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36'; $headers[]= 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8'; $headers[]= 'Referer: http://www.streamlive.to/tv-guide'; $headers[]= 'Accept-Encoding: gzip, deflate, sdch'; $headers[]= 'Accept-Language: en-US,en;q=0.8'; $headers[]= 'Cookie: PHPSESSID=f7mqqpgm3beto9a462fn8a2pl3; _gat=1; _ga=GA1.2.1850053691.1432824439'; $ch = curl_init('http://www.streamlive.to/tv-ajax.php'); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_COOKIESESSION, true); curl_setopt($ch, CURLOPT_FRESH_CONNECT, true); //curl_setopt($ch, CURLOPT_HEADER,true); curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); curl_setopt($ch, CURLOPT_VERBOSE, true);//0-FALSE 1 TRUE curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, 'action=getchannelsched&id=14'); $html = curl_exec($ch); $html = gzdecode($html); curl_close($ch); echo '<pre>'; var_dump($html); echo '</pre>'; if (file_put_contents ('data.txt', $html) !== false) { echo 'Success!'; } else { echo 'Failed'; } Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512891 Share on other sites More sharing options...
Solution jcbones Posted May 31, 2015 Solution Share Posted May 31, 2015 This is the result: Chromium**** <div class="row thead"> <input class="fullgrid_btn" type="button" value="View All Channels" /> Daily Programming on <span class="channel_name">BRAVO</span> for Sunday, May 31, 2015 thru Sunday, June 7, 2015 </div> <div class="tbody"> <div class="row"> <div class="col th"> 12:00 AM <br/> May 31, 2015 </div> <div class="prog_cols" > <div class="col ts prog_910899 movies" data-progid="910899" data-catid="movies" > <span class="prog_name">Enough</span> <div class="prog_time">May 31, 2015, 12:00 am - 2:00 am</div> <a class="btn_watchlist disabled" href="javascript:void(0)" data-progid="910899">(+) add to watchlist</a> <div class="prog_desc">A woman on the lam with her young daughter takes drastic steps when her abusive husband tracks them down.</div> </div> </div> <a class="watchnow" href="http://www.streamlive.to/premium">Watch Now</a> </div> <div class="row"> <div class="col th"> 2:00 AM <br/> May 31, 2015 </div> <div class="prog_cols" > <div class="col ts prog_910900 " data-progid="910900" data-catid="" > <span class="prog_name">The Real Housewives of Atlanta</span> <div class="prog_time">May 31, 2015, 2:00 am - 3:00 am</div> <a class="btn_watchlist disabled" href="javascript:void(0)" data-progid="910900">(+) add to watchlist</a> <div class="prog_desc">Kandi Burruss and Todd Tucker plan a family ski trip to Colorado. </div> </div> </div> <a class="watchnow" href="http://www.streamlive.to/premium">Watch Now</a> </div> <div class="row"> <div class="col th"> 3:00 AM <br/> May 31, 2015 </div> <div class="prog_cols" > <div class="col ts prog_910901 " data-progid="910901" data-catid="" > <span class="prog_name">Blood, Sweat & Heels</span> <div class="prog_time">May 31, 2015, 3:00 am - 4:00 am</div> <a class="btn_watchlist disabled" href="javascript:void(0)" data-progid="910901">(+) add to watchlist</a> <div class="prog_desc">Chantelle has a fast-paced blind date with a sightly bachelor while Mica has an intimate date at her apartment. In other developments, Demetria's cake-tasting gathering takes a sour turn; the crew heads to the Hamptons; and a painful moment tests Daisy.</div> </div> </div> <a class="watchnow" href="http://www.streamlive.to/premium">Watch Now</a> </div> <div class="row"> <div class="col th"> 4:00 AM <br/> May 31, 2015 </div> <div class="prog_cols" > <div class="col ts prog_910902 " data-progid="910902" data-catid="" > <span class="prog_name">Paid Programming</span> <div class="prog_time">May 31, 2015, 4:00 am - 6:00 am</div> <a class="btn_watchlist disabled" href="javascript:void(0)" data-progid="910902">(+) add to watchlist</a> <div class="prog_desc">No details are there for this program.</div> </div> </div> <a class="watchnow" href="http://www.streamlive.to/premium">Watch Now</a> </div> <div class="row"> <div class="col th"> 6:00 AM <br/> May 31, 2015 </div> <div class="prog_cols" > <div class="col ts prog_910903 " data-progid="910903" data-catid="" > <span class="prog_name">Million Dollar Listing New York</span> <div class="prog_time">May 31, 2015, 6:00 am - 8:00 am</div> <a class="btn_watchlist disabled" href="javascript:void(0)" data-progid="910903">(+) add to watchlist</a> <div class="prog_desc">Fredrik's reputation is tested when developers threaten to pull out of premium finishes they promised. Elsewhere, Ryan reunites with a familiar blonde, and Luis pursues one of the city's top developers.</div> </div> </div> <a class="watchnow" href="http://www.streamlive.to/premium">Watch Now</a> </div> <div class="row"> ***Truncated for posting*** Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512892 Share on other sites More sharing options...
codeinphp Posted May 31, 2015 Author Share Posted May 31, 2015 Thanks again. I will go thru your code and find my error. When I run your script I get a proper var_dump. Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512893 Share on other sites More sharing options...
codeinphp Posted May 31, 2015 Author Share Posted May 31, 2015 What does your data.txt file look like? My result to browser is correct but nothing is saved to file. Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512894 Share on other sites More sharing options...
codeinphp Posted May 31, 2015 Author Share Posted May 31, 2015 Never mind, my mistake. I am getting info back. Once again thank you for taking your time to help me. Quote Link to comment https://forums.phpfreaks.com/topic/296552-command-line-curl-and-php-curl-different-results/#findComment-1512895 Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.