Jump to content

Recommended Posts

I'm trying to login to a site, navigate a couple of pages deep, and then scrape a table.  I had all this working, but the site changed their login workflow...and I can't figure out what I need to do to make this work.

The form used in a browser to login has itself as the action page.  In a browser, you are automatically redirected to another page, that puts a random get variable at the end of the URL.

Here's the main login page:  https://www.premiereaffidavits.com/scripts/cgiip.exe/WService=webprime/login.html

When I run a standard curl login code...I only get the login page returned to me.  How can I get the redirected page (including the random GET variable at the end of the URL) that you get using a browser?  I see no javascript, or other ways for the action page to redirect to the second page you get in a browser.  What am I missing here?

I can generate login info if you need it.

Thanks.

Edited by bschultz
Link to post
Share on other sites
<?php

$username = 'xxx';
$password = 'xxx';
$postdata = '?username='.$username.'&password='.$password;

$url = 'https://www.premiereaffidavits.com/scripts/cgiip.exe/WService=webprime/login.html';

$ch = curl_init(); 
curl_setopt ($ch, CURLOPT_URL, $url); 
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE); 
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"); 
curl_setopt ($ch, CURLOPT_TIMEOUT, 60); 
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1); 
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); 
curl_setopt ($ch, CURLOPT_REFERER, $url); 
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postdata); 
curl_setopt ($ch, CURLOPT_POST, 1); 
$result = curl_exec ($ch); 

echo $result;  

curl_close($ch);
?>

This only returns the login page...not the redirect page

Link to post
Share on other sites

You need to send the name of the button also.

Use the browsers dev tools or something like Fiddler to inspect all the requests being made and re-create the ones you need as closely as possible.

 

Edited by kicken
Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.