Jump to content


Photo

Use CURL to read website and load fields? website automation?


  • Please log in to reply
5 replies to this topic

#1 cgchris99

cgchris99
  • Members
  • PipPipPip
  • Advanced Member
  • 37 posts

Posted 10 October 2006 - 03:19 PM

Should I use CURL or wget to website info and preload the necessary fields?

I have never done this type of programming (ie CURL/wget)

I need it to access the site, log me in and go to a certain page and extract the information and store it  in my MySQL table.

Are there any examples on how to code something like this. Not the table work but the CURL/wget and filling in the fields.

Thanks for any advice.


#2 michaellunsford

michaellunsford
  • Members
  • PipPipPip
  • Advanced Member
  • 1,023 posts
  • LocationLouisiana, USA

Posted 10 October 2006 - 03:56 PM

here's the first part from the manuel -- some additions to help you along. You'll probably need to make one post for the username, password, etc. Then make a second request for your data. The "CURLOPT_RETURNTRANSFER" part says you want to return the page queried to a variable. If you're a good curler, you can probably figure it out pretty quickly with the following example and links:

example from: http://usphp.com/man...n.curl-exec.php
all curl options: http://usphp.com/curl_setopt

<?php
// create a new curl resource
$ch = curl_init();

// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "http://www.example.com/"); //full URL of your query
curl_setopt($ch, CURLOPT_HEADER, 0); //don't display or store the header information returned
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //curl_exec() will returns the page instead of true or false
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt"); //store incoming cookies to this file
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookiex.txt"); //retrieve cookies asked for by the site from this file
curl_setopt($ch, CURLOPT_POST, 1); //we're using the "POST" method
curl_setopt($ch, CURLOPT_POSTFIELDS, "user=myusername&password=mypassword"); //post these fields

// grab URL and pass it to the browser
$var = curl_exec($ch);

// close curl resource, and free up system resources
curl_close($ch);

echo $var;
?>


#3 cgchris99

cgchris99
  • Members
  • PipPipPip
  • Advanced Member
  • 37 posts

Posted 10 October 2006 - 06:23 PM

thanks for the code and the links.  I'll do some testing tonight.


#4 cgchris99

cgchris99
  • Members
  • PipPipPip
  • Advanced Member
  • 37 posts

Posted 11 October 2006 - 12:47 AM

How do I get curl to execute the javascript submit on the form?

This is the code....
alt="Member Login" onclick="doSubmit('Login_loginbutton')"



#5 michaellunsford

michaellunsford
  • Members
  • PipPipPip
  • Advanced Member
  • 1,023 posts
  • LocationLouisiana, USA

Posted 11 October 2006 - 12:53 AM

You have to read the javascript and see what it does. More than likely, it just makes sure the form has been filled in properly. Just make sure you're posting it to the same place "action" goes to (if there isn't an action, just post it to the starting page).

#6 cgchris99

cgchris99
  • Members
  • PipPipPip
  • Advanced Member
  • 37 posts

Posted 11 October 2006 - 04:26 PM

thanks.  I figured out what the post did via the javascript.  I think it is working fine now.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users