Jump to content


This topic is now archived and is closed to further replies.


Use CURL to read website and load fields? website automation?

Recommended Posts

Should I use CURL or wget to website info and preload the necessary fields?

I have never done this type of programming (ie CURL/wget)

I need it to access the site, log me in and go to a certain page and extract the information and store it  in my MySQL table.

Are there any examples on how to code something like this. Not the table work but the CURL/wget and filling in the fields.

Thanks for any advice.

Share this post

Link to post
Share on other sites
here's the first part from the manuel -- some additions to help you along. You'll probably need to make one post for the username, password, etc. Then make a second request for your data. The "CURLOPT_RETURNTRANSFER" part says you want to return the page queried to a variable. If you're a good curler, you can probably figure it out pretty quickly with the following example and links:

example from: http://usphp.com/manual/en/function.curl-exec.php
all curl options: http://usphp.com/curl_setopt

// create a new curl resource
$ch = curl_init();

// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "http://www.example.com/"); //full URL of your query
curl_setopt($ch, CURLOPT_HEADER, 0); //don't display or store the header information returned
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //curl_exec() will returns the page instead of true or false
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt"); //store incoming cookies to this file
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookiex.txt"); //retrieve cookies asked for by the site from this file
curl_setopt($ch, CURLOPT_POST, 1); //we're using the "POST" method
curl_setopt($ch, CURLOPT_POSTFIELDS, "user=myusername&password=mypassword"); //post these fields

// grab URL and pass it to the browser
$var = curl_exec($ch);

// close curl resource, and free up system resources

echo $var;

Share this post

Link to post
Share on other sites
How do I get curl to execute the javascript submit on the form?

This is the code....
alt="Member Login" onclick="doSubmit('Login_loginbutton')"

Share this post

Link to post
Share on other sites
You have to read the javascript and see what it does. More than likely, it just makes sure the form has been filled in properly. Just make sure you're posting it to the same place "action" goes to (if there isn't an action, just post it to the starting page).

Share this post

Link to post
Share on other sites


Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.