Jump to content


Photo

curl and cookies in php


  • Please log in to reply
26 replies to this topic

#1 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 19 December 2013 - 02:38 PM

I want to ask you how to send headers to simulate the cookies which keeps values of options and then to download the file. For example

 

If I would like to insert IP here:

http://www.iplocation.net/index.php

and then download the file how to do it?

 

I have read one article, where is this code

http://paste.ofcode....ZHKxk3VUmZcB4xe

But it did not anything.

 

Also I have tried to change the above code to this:

http://paste.ofcode....yhPESkUHD9VLedf

to test the options saved in cookies. But nothing works (no data displayed and no file downloaded). What I do wrong?



#2 scootstah

scootstah

    Advanced Member

  • Gurus
  • 2,889 posts
  • LocationUSA

Posted 19 December 2013 - 03:12 PM

Those cookies are Google Analytics tracking cookies; you don't need to send them in your cURL request.

It looks like all you need to do is send a POST request with the following data:
query=127.0.0.1&submit=Query
Replace 127.0.0.1 with the IP that you want.
while(!$succeed = try());

#3 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 19 December 2013 - 05:28 PM

Those cookies are Google Analytics tracking cookies; you don't need to send them in your cURL request.

It looks like all you need to do is send a POST request with the following data:

query=127.0.0.1&submit=Query
Replace 127.0.0.1 with the IP that you want.

 

 

It seems like you don't understand what I want. I need to learn to use cookies in CURL but this is example page, where I wanted to test it. Also the other codes I tried did not work at all. Where is the problem? I want to make my codes working and do what is expected.


Edited by murfy, 19 December 2013 - 05:30 PM.


#4 jazzman1

jazzman1

    Advanced Member

  • Gurus
  • 2,663 posts
  • LocationMississauga, Canada

Posted 20 December 2013 - 01:52 PM

Do you know how to send POST or GET data filling input fields on the html form in cURL? To get the cookies from the server you shoud use CURLOPT_COOKIEFILE to read cookie and CURLOPT_COOKIEJAR to write down the cookie on the file. Also, you have to show us how you desighed the server side code :) Do you want to give you few examples of that? 


Edited by jazzman1, 20 December 2013 - 01:53 PM.


#5 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 20 December 2013 - 02:43 PM

jazzman1:

It was to me explained that there are two codes doing the same thing. So I understood that I must to try them separated. So they worked. And now I tried to use curl to send a form data and download page. I have succeeded to do that. But now I would like to do the same with php and here I fail. Enter some IP you want to locate:

$context = stream_context_create(array('http' => array(
'method' => 'POST',
'header' => 'Content-Type: application/x-www-form-urlencoded',
'content' => http_build_query(array('query' => '71.44.87.55'))
)));
$fp = fopen("http://www.iplocation.net/index.php", "r", false, $context);
$meta = stream_get_meta_data($fp);
$headers = $meta["wrapper_data"];
print_r($headers);
$content=fread($fp, 8000);
fclose($fp);
echo $content;

It waited too long so nothing downloaded. What's wrong?


Edited by murfy, 20 December 2013 - 02:45 PM.


#6 jazzman1

jazzman1

    Advanced Member

  • Gurus
  • 2,663 posts
  • LocationMississauga, Canada

Posted 20 December 2013 - 03:00 PM

 

It waited too long so nothing downloaded. What's wrong?

 

Nothing, it works just fine to me.

 

Do you have php's error_reporting set to E_ALL and display_errors set to ON so that any php detected errors will be reported and displayed?

 

EDIT:

 

Try,

<?php
ini_set('display_errors',1);
ini_set('display_startup_errors',1);
ini_set('output_buffering','Off');
error_reporting(-1);
$context = stream_context_create(array('http' => array(
'method' => 'POST',
'header' => 'Content-Type: application/x-www-form-urlencoded',
'content' => http_build_query(array('query' => '71.44.87.55'))
)));
$fp = fopen("http://www.iplocation.net/index.php", "r", false, $context);
$meta = stream_get_meta_data($fp);
$headers = $meta["wrapper_data"];
print_r($headers);
$content=fread($fp, 8000);
fclose($fp);
echo $content;

Edited by jazzman1, 20 December 2013 - 03:05 PM.


#7 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 21 December 2013 - 03:51 AM

U R right. It works. But how can I do filesize messuring? 160 kB is still pretty short.

$url="http://www.iplocation.net/index.php";
$fp = fopen($url, "r", false, $context);
$meta = stream_get_meta_data($fp);
$content=fread($fp, filesize($url));

fread obtains 0 length on parameter 2.

 

The command stream_get_meta_data() is always neccessary to perform before I can read th file?

 

Looks like $content=fread($fp, 500000); still downloads only 5kB of data, don't you thing? But table with location should be downloaded.


Edited by murfy, 21 December 2013 - 03:57 AM.


#8 jazzman1

jazzman1

    Advanced Member

  • Gurus
  • 2,663 posts
  • LocationMississauga, Canada

Posted 21 December 2013 - 07:06 AM

Try,

ini_set('display_errors',1);
ini_set('display_startup_errors',1);
ini_set('output_buffering','Off');
error_reporting(-1);
$context = stream_context_create(array('http' => array(
'method' => 'POST',
'header' => 'Content-Type: application/x-www-form-urlencoded',
'content' => http_build_query(array('query' => '71.44.87.55'))
)));
$fp = fopen("http://www.iplocation.net/index.php", "r", false, $context);
$meta = stream_get_meta_data($fp);
$headers = $meta["wrapper_data"];
//print_r($headers);
$content = '';
while(!feof($fp)) {
   $content = fread($fp, 4096);
   echo $content;
}
fclose($fp);

If you want to get (use) all css, js, pictures, etc, from their web page, set the base html tag inside the head tag of the document. 

For example:

<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>IP Location Finder - Geolocation</title>
<meta name="keywords" content="ip, ip address, find, find ip, location, router, dns, reverse, tcp/ip, lookup, traffic, traffic analysis, your address, dynamic, change, isp, internet, provider, inet, net, localhost">
<meta name="description" content="How do I find a geographical location of an IP address?">
<base href="http://www.iplocation.net/"> 
<link rel="stylesheet" href="./style.css" type="text/css">
<link rel="SHORTCUT ICON" href="./favicon.ico"> 


#9 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 21 December 2013 - 10:33 AM

Nice. So basicly all what is needed is to set the loop? And the n bytes number in the fread should be 4096? It cannot be bigger?



#10 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 21 December 2013 - 01:22 PM

BTW: I have messured the time how long it taken to download the file:

with curl action taken 3.12357401848 s, with php it takes just 1.59751701355 s. PHP download looks 2x faster then curl.



#11 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 21 December 2013 - 01:56 PM

Also I would like to ask you how to send request simulation input checked but the inputs have all same name:

<input type="checkbox" checked="checked" value="0" name="pr[]">
<input type="checkbox" checked="checked" value="1" name="pr[]">
<input type="checkbox" checked="checked" value="2" name="pr[]">
<input id="ports" type="checkbox" checked="">

How to send that I want first checkbox check, second and third not checked, next checked?



#12 jazzman1

jazzman1

    Advanced Member

  • Gurus
  • 2,663 posts
  • LocationMississauga, Canada

Posted 21 December 2013 - 02:53 PM

Hey murfy,

without going into too much details I wanna tell you that fopen and curl are two different tools desighed to do different tasks.
For instance, cURL is a library in php for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP, POP3S, RTMP, RTSP, SCP, SFTP, SMTP and so forth protocols, a lot of these protocols are not supported by fopen at least is not very suitable to handle the data with fopen or other similar tools.
On the other hand, fopen is a tool desighed to open files or URL's from remote distance, so if you have a big video file in the remote
machine and want to write down on the local one you should better use fopen and fread to chunk the file into some parts writting it in the local machine.
Based on your example I see, you want to grab a web page into a string, so to increase the speed a little more you could use the file_get_contents function instead of fopen or curl. It would be similar like:

$context = stream_context_create(array('http' => array(
'method' => 'POST',
'header' => 'Content-Type: application/x-www-form-urlencoded',
'content' => http_build_query(array('query' => '71.44.87.55'))
)));

// Open the file using the HTTP headers set above
$file = file_get_contents('http://www.iplocation.net/index.php', false, $context);

echo $file;

 

Also I would like to ask you how to send request simulation input checked but the inputs have all same name:

 

I am not entirly sure what, exactly you want to do here but.....it sounds you should use cURL to login into the web page and set check attributes to these checkboxes you want to be checked.


Edited by jazzman1, 21 December 2013 - 02:56 PM.


#13 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 21 December 2013 - 04:14 PM

I tried this code, but when I check headers, I see there is only GET request. Not the post.

 

http://paste.ofcode....XZpfh64b3g72ax2

 

Why?



#14 jazzman1

jazzman1

    Advanced Member

  • Gurus
  • 2,663 posts
  • LocationMississauga, Canada

Posted 21 December 2013 - 05:05 PM

Look at the examples of php.net. Maybe you're doing something wrong or at least don't undestand something very well.



#15 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 22 December 2013 - 04:22 AM

Is it possible to add next headers?

 

I try this:

http://paste.ofcode....vj8eAraLFkudGkm

 

Resulting with error:

Warning: fopen - failed to open stream: Redirection limit reached, aborting ...

 

This is second time when I have this error. What does it mean?



#16 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 22 December 2013 - 05:22 AM

Here is my complete code. It should send/submit the form with some of the checkboxes unchecked (this is filtration to limit cound of the items displayed on the site) and recieve one page only.

 

http://paste.ofcode....642CU5p2mxSFkX6

Here is options to the options file:

http://paste.ofcode....3bmbC7vTS5c5Q6r

 

However the request is not good. It should change host name. Do you know how to correct this?



#17 jazzman1

jazzman1

    Advanced Member

  • Gurus
  • 2,663 posts
  • LocationMississauga, Canada

Posted 22 December 2013 - 09:45 AM

Do var_dump(http_build_query( $options)) and post the output.



#18 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 22 December 2013 - 11:17 AM

Do you mean post it to you or post it as a content in http request?

http://paste.ofcode....fLfeADZKQxK37TV



#19 jazzman1

jazzman1

    Advanced Member

  • Gurus
  • 2,663 posts
  • LocationMississauga, Canada

Posted 22 December 2013 - 01:25 PM

Did you try to set all requested header inside context header?

it's mine that I've got from their domain:

 

 

Accept    text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding    gzip, deflate
Accept-Language    en-US,en;q=0.5
Cache-Control    no-cache
Connection    keep-alive
Content-Length    3
Content-Type    application/x-www-form-urlencoded; charset=UTF-8
Cookie    PHPSESSID=f7nmtt0eukn3efqjust9k4k2f1; __utma=82459535.870087495.1387735198.1387735198.1387735198.1; __utmb=82459535.3.10.1387735198; __utmc=82459535; __utmz=82459535.1387735198.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
Host    hidemyass.com
Pragma    no-cache
Referer    http://hidemyass.com...t/search-225892
User-Agent    Mozilla/5.0 (X11; Debian; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0



#20 murfy

murfy

    Advanced Member

  • Members
  • PipPipPip
  • 48 posts

Posted 22 December 2013 - 04:43 PM

I added the additional headers in  $addHeaders : host and reffererer . Also the user agent is set.

exactly as you see it in the link.

 

My headers as I copy them from the firebug - net card:

http://paste.ofcode....brbRyEkAwmW4Zxc

 

Notice, that there is localhost...

                               

Edited by murfy, 22 December 2013 - 04:45 PM.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users

Cheap Linux VPS from $5
SSD Storage, 30 day Guarantee
1 TB of BW, 100% Network Uptime

AlphaBit.com