Connect To Secure Site With Cookies, cURL

[Tanner8]

Hey, a secure site I am trying to access only allows you to with cookies enabled. The thing is, it still doesn't work with cURL, what is the best way I can emulate a browser? Here is my current code, which does not work, it is giving me the result as if it doesn't accept cookies. What's wrong?

 

$url="https://www.site.com/test.php?boe=325
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']); 
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIESESSION, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
$fullPage= curl_exec($ch);
curl_close($ch);

 

Thanks

[cags]

That appears to be the right idea with CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE, the problem you are probably encountering is that it expects you to already have the cookie when you arrive at that page. This is a common issue with ASPX websites. Given an ASPX page which has a search form, in order to perform a search you need to scrape the page in order to gain a session cookie (as well as scrape some other data from it), you then need to perform another curl_exec to submit the search. Contrast that to a standard PHP where you can simply jump straight in and post the data. It's likely this kind of issue you are running into (even if the target site isn't ASPX).

[Tanner8]

That appears to be the right idea with CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE, the problem you are probably encountering is that it expects you to already have the cookie when you arrive at that page. This is a common issue with ASPX websites. Given an ASPX page which has a search form, in order to perform a search you need to scrape the page in order to gain a session cookie (as well as scrape some other data from it), you then need to perform another curl_exec to submit the search. Contrast that to a standard PHP where you can simply jump straight in and post the data. It's likely this kind of issue you are running into (even if the target site isn't ASPX).

 

That's actually exactly what I am doing.  The first page I go to I am grabbing the cookie. Then I set it before I try to go to my new page, and it still isn't letting me. I even took out the cookiefile part of the second cURL so we aren't using that one. Hmm

[cags]

Is it an ASPX page?

[Tanner8]

Some of the pages end in the suffix .asp, others end in .jsp, and others end in .do. The one I need to access ends in .do.

[cags]

As previously mentioned there are far more values you need for form submission on an ASP page so if one of the processes you are performing with cURL is submission of a form then that could be the reason. I think .do are Java files but never used them so not sure what would be involved with that.

[Tanner8]

As previously mentioned there are far more values you need for form submission on an ASP page so if one of the processes you are performing with cURL is submission of a form then that could be the reason. I think .do are Java files but never used them so not sure what would be involved with that.

 

Ya most of this seems to be in java, because when you F up, you get java errors. So you don't know how to allow cookies to work with java? I hope someone does, I needa get this done!

 

Edit: Ok it seems like the cookie.txt isn't getting all of the cookies. I am going through it with http live headers and the cooke seems much larger then what is in the file. In the file it pretty much just stores the JSessionID, not the utma stuff.