curl and cookies in php

[murfy]

I want to ask you how to send headers to simulate the cookies which keeps values of options and then to download the file. For example

 

If I would like to insert IP here:

http://www.iplocation.net/index.php

and then download the file how to do it?

 

I have read one article, where is this code

http://paste.ofcode.org/bfFL99ZZHKxk3VUmZcB4xe

But it did not anything.

 

Also I have tried to change the above code to this:

http://paste.ofcode.org/JwSvGFPyhPESkUHD9VLedf

to test the options saved in cookies. But nothing works (no data displayed and no file downloaded). What I do wrong?

[scootstah]

Those cookies are Google Analytics tracking cookies; you don't need to send them in your cURL request.

 

It looks like all you need to do is send a POST request with the following data:

query=127.0.0.1&submit=Query

Replace 127.0.0.1 with the IP that you want.

[murfy]

Those cookies are Google Analytics tracking cookies; you don't need to send them in your cURL request.

 

It looks like all you need to do is send a POST request with the following data:

query=127.0.0.1&submit=Query

Replace 127.0.0.1 with the IP that you want.

 

 

It seems like you don't understand what I want. I need to learn to use cookies in CURL but this is example page, where I wanted to test it. Also the other codes I tried did not work at all. Where is the problem? I want to make my codes working and do what is expected.

Edited December 19, 2013 by murfy

[jazzman1]

Do you know how to send POST or GET data filling input fields on the html form in cURL? To get the cookies from the server you shoud use CURLOPT_COOKIEFILE to read cookie and CURLOPT_COOKIEJAR to write down the cookie on the file. Also, you have to show us how you desighed the server side code Do you want to give you few examples of that? 

Edited December 20, 2013 by jazzman1

[murfy]

jazzman1:

It was to me explained that there are two codes doing the same thing. So I understood that I must to try them separated. So they worked. And now I tried to use curl to send a form data and download page. I have succeeded to do that. But now I would like to do the same with php and here I fail. Enter some IP you want to locate:

$context = stream_context_create(array('http' => array(
'method' => 'POST',
'header' => 'Content-Type: application/x-www-form-urlencoded',
'content' => http_build_query(array('query' => '71.44.87.55'))
)));
$fp = fopen("http://www.iplocation.net/index.php", "r", false, $context);
$meta = stream_get_meta_data($fp);
$headers = $meta["wrapper_data"];
print_r($headers);
$content=fread($fp, 8000);
fclose($fp);
echo $content;

It waited too long so nothing downloaded. What's wrong?

Edited December 20, 2013 by murfy

[jazzman1]

 

It waited too long so nothing downloaded. What's wrong?

 

Nothing, it works just fine to me.

 

Do you have php's error_reporting set to E_ALL and display_errors set to ON so that any php detected errors will be reported and displayed?

 

EDIT:

 

Try,

<?php
ini_set('display_errors',1);
ini_set('display_startup_errors',1);
ini_set('output_buffering','Off');
error_reporting(-1);
$context = stream_context_create(array('http' => array(
'method' => 'POST',
'header' => 'Content-Type: application/x-www-form-urlencoded',
'content' => http_build_query(array('query' => '71.44.87.55'))
)));
$fp = fopen("http://www.iplocation.net/index.php", "r", false, $context);
$meta = stream_get_meta_data($fp);
$headers = $meta["wrapper_data"];
print_r($headers);
$content=fread($fp, 8000);
fclose($fp);
echo $content;

Edited December 20, 2013 by jazzman1

[murfy]

U R right. It works. But how can I do filesize messuring? 160 kB is still pretty short.

$url="http://www.iplocation.net/index.php";
$fp = fopen($url, "r", false, $context);
$meta = stream_get_meta_data($fp);
$content=fread($fp, filesize($url));

fread obtains 0 length on parameter 2.

 

The command stream_get_meta_data() is always neccessary to perform before I can read th file?

 

Looks like $content=fread($fp, 500000); still downloads only 5kB of data, don't you thing? But table with location should be downloaded.

Edited December 21, 2013 by murfy

[jazzman1]

Try,

ini_set('display_errors',1);
ini_set('display_startup_errors',1);
ini_set('output_buffering','Off');
error_reporting(-1);
$context = stream_context_create(array('http' => array(
'method' => 'POST',
'header' => 'Content-Type: application/x-www-form-urlencoded',
'content' => http_build_query(array('query' => '71.44.87.55'))
)));
$fp = fopen("http://www.iplocation.net/index.php", "r", false, $context);
$meta = stream_get_meta_data($fp);
$headers = $meta["wrapper_data"];
//print_r($headers);
$content = '';
while(!feof($fp)) {
   $content = fread($fp, 4096);
   echo $content;
}
fclose($fp);

If you want to get (use) all css, js, pictures, etc, from their web page, set the base html tag inside the head tag of the document. 

For example:

<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>IP Location Finder - Geolocation</title>
<meta name="keywords" content="ip, ip address, find, find ip, location, router, dns, reverse, tcp/ip, lookup, traffic, traffic analysis, your address, dynamic, change, isp, internet, provider, inet, net, localhost">
<meta name="description" content="How do I find a geographical location of an IP address?">
<base href="http://www.iplocation.net/"> 
<link rel="stylesheet" href="./style.css" type="text/css">
<link rel="SHORTCUT ICON" href="./favicon.ico"> 

[murfy]

Nice. So basicly all what is needed is to set the loop? And the n bytes number in the fread should be 4096? It cannot be bigger?

[murfy]

BTW: I have messured the time how long it taken to download the file:

with curl action taken 3.12357401848 s, with php it takes just 1.59751701355 s. PHP download looks 2x faster then curl.

[murfy]

Also I would like to ask you how to send request simulation input checked but the inputs have all same name:

<input type="checkbox" checked="checked" value="0" name="pr[]">
<input type="checkbox" checked="checked" value="1" name="pr[]">
<input type="checkbox" checked="checked" value="2" name="pr[]">
<input id="ports" type="checkbox" checked="">

How to send that I want first checkbox check, second and third not checked, next checked?

[jazzman1]

Hey murfy,

without going into too much details I wanna tell you that fopen and curl are two different tools desighed to do different tasks.
For instance, cURL is a library in php for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP, POP3S, RTMP, RTSP, SCP, SFTP, SMTP and so forth protocols, a lot of these protocols are not supported by fopen at least is not very suitable to handle the data with fopen or other similar tools.
On the other hand, fopen is a tool desighed to open files or URL's from remote distance, so if you have a big video file in the remote
machine and want to write down on the local one you should better use fopen and fread to chunk the file into some parts writting it in the local machine.
Based on your example I see, you want to grab a web page into a string, so to increase the speed a little more you could use the file_get_contents function instead of fopen or curl. It would be similar like:

$context = stream_context_create(array('http' => array(
'method' => 'POST',
'header' => 'Content-Type: application/x-www-form-urlencoded',
'content' => http_build_query(array('query' => '71.44.87.55'))
)));

// Open the file using the HTTP headers set above
$file = file_get_contents('http://www.iplocation.net/index.php', false, $context);

echo $file;

 

Also I would like to ask you how to send request simulation input checked but the inputs have all same name:

 

I am not entirly sure what, exactly you want to do here but.....it sounds you should use cURL to login into the web page and set check attributes to these checkboxes you want to be checked.

Edited December 21, 2013 by jazzman1

[murfy]

I tried this code, but when I check headers, I see there is only GET request. Not the post.

 

http://paste.ofcode.org/Et2dERYXZpfh64b3g72ax2

 

Why?

[jazzman1]

Look at the examples of php.net. Maybe you're doing something wrong or at least don't undestand something very well.

[murfy]

Is it possible to add next headers?

 

I try this:

http://paste.ofcode.org/RHfCqLWvj8eAraLFkudGkm

 

Resulting with error:

Warning: fopen - failed to open stream: Redirection limit reached, aborting ...

 

This is second time when I have this error. What does it mean?

[murfy]

Here is my complete code. It should send/submit the form with some of the checkboxes unchecked (this is filtration to limit cound of the items displayed on the site) and recieve one page only.

 

http://paste.ofcode.org/Jxgcaie642CU5p2mxSFkX6

Here is options to the options file:

http://paste.ofcode.org/8pgNrZV3bmbC7vTS5c5Q6r

 

However the request is not good. It should change host name. Do you know how to correct this?

[jazzman1]

Do var_dump(http_build_query( $options)) and post the output.

[murfy]

Do you mean post it to you or post it as a content in http request?

http://paste.ofcode.org/dmUqvzvfLfeADZKQxK37TV

[jazzman1]

Did you try to set all requested header inside context header?

it's mine that I've got from their domain:

 

 

Accept    text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding    gzip, deflate
Accept-Language    en-US,en;q=0.5
Cache-Control    no-cache
Connection    keep-alive
Content-Length    3
Content-Type    application/x-www-form-urlencoded; charset=UTF-8
Cookie    PHPSESSID=f7nmtt0eukn3efqjust9k4k2f1; __utma=82459535.870087495.1387735198.1387735198.1387735198.1; __utmb=82459535.3.10.1387735198; __utmc=82459535; __utmz=82459535.1387735198.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Host    hidemyass.com
Pragma    no-cache
Referer    http://hidemyass.com/proxy-list/search-225892
User-Agent    Mozilla/5.0 (X11; Debian; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0

[murfy]

I added the additional headers in  $addHeaders : host and reffererer . Also the user agent is set.

exactly as you see it in the link.

 

My headers as I copy them from the firebug - net card:

http://paste.ofcode.org/G8TpiKLbrbRyEkAwmW4Zxc

 

Notice, that there is localhost...

                                Edited December 22, 2013 by murfy

[jazzman1]

Hey murfy,

sorry for the delay but there was a big ice storm in Toronto and GTA which left my city dealing with widespread power outages and transit problems for more than 15 hours.

So, your problem is, that the post form action doesn't execute any data to the remote URL destination, even though you have a set of cookies and proper header information, right?

I will try to test your script by myself, if I have the time today and I'll back with new reply if I got a success result. If it still doesn't work the easiest way is to use cURL of course  

[murfy]

Hi jazzman1

I had a break from tomorrow up today, so maybe I miss something. The posted header was not the look as I would expected. I have seen host: localhost instead host:hidemyass.com. But I am not sure if php works with the cookies. I don't see there any cookie header in the context. But the server really needs cookies because it will not send the filtered results if I disable cookies and the options will reset. Maybe curl is only change, but then it will be twice so slower.

[jazzman1]

Twice slower?  I believe not. 

 

Take this one and test the speed, it's a part of my script

//$data = array('name'=>'userName','password'=>'userPass','Submit'=>'OK');

$curl_cookie = "cookies.txt";

//$fp = fopen("example_homepage.txt", "w");

// create a new cURL resource
$ch = curl_init();

// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, 'http://hidemyass.com/proxy-list/search-225892');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, FALSE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $curl_cookie); // Read cookie file
curl_setopt($ch, CURLOPT_COOKIEJAR,  $curl_cookie); // Write cookie file
//curl_setopt($ch, CURLOPT_POSTFIELDS, $data); // set the post data
//curl_setopt($ch, CURLOPT_FILE, $fp);  // write the data in external file
// grab URL and pass it to the browser
$output = curl_exec($ch);

$info = curl_getinfo($ch);

curl_close($ch);

//fclose($fp);

Edited December 23, 2013 by jazzman1

[murfy]

two warnings:

curl_setopt() [function.curl-setopt]: CURLOPT_FOLLOWLOCATION cannot be activated when in safe_mode or an open_basedir is set

curl_setopt() [function.curl-setopt]: Unable to access cookies.txt

 

Time is

1.11 - 1.25 for 50 items.

[jazzman1]

Check the permissions to this working directory. I'm on CentOS and Debian linux distros.