cgchris99 Posted October 10, 2006 Share Posted October 10, 2006 Should I use CURL or wget to website info and preload the necessary fields? I have never done this type of programming (ie CURL/wget)I need it to access the site, log me in and go to a certain page and extract the information and store it in my MySQL table.Are there any examples on how to code something like this. Not the table work but the CURL/wget and filling in the fields.Thanks for any advice. Quote Link to comment Share on other sites More sharing options...
michaellunsford Posted October 10, 2006 Share Posted October 10, 2006 here's the first part from the manuel -- some additions to help you along. You'll probably need to make one post for the username, password, etc. Then make a second request for your data. The "CURLOPT_RETURNTRANSFER" part says you want to return the page queried to a variable. If you're a good curler, you can probably figure it out pretty quickly with the following example and links:example from: http://usphp.com/manual/en/function.curl-exec.phpall curl options: http://usphp.com/curl_setopt[code=php:0]<?php// create a new curl resource$ch = curl_init();// set URL and other appropriate optionscurl_setopt($ch, CURLOPT_URL, "http://www.example.com/"); //full URL of your querycurl_setopt($ch, CURLOPT_HEADER, 0); //don't display or store the header information returnedcurl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //curl_exec() will returns the page instead of true or falsecurl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt"); //store incoming cookies to this filecurl_setopt($ch, CURLOPT_COOKIEFILE, "cookiex.txt"); //retrieve cookies asked for by the site from this filecurl_setopt($ch, CURLOPT_POST, 1); //we're using the "POST" methodcurl_setopt($ch, CURLOPT_POSTFIELDS, "user=myusername&password=mypassword"); //post these fields// grab URL and pass it to the browser$var = curl_exec($ch);// close curl resource, and free up system resourcescurl_close($ch);echo $var;?>[/code] Quote Link to comment Share on other sites More sharing options...
cgchris99 Posted October 10, 2006 Author Share Posted October 10, 2006 thanks for the code and the links. I'll do some testing tonight. Quote Link to comment Share on other sites More sharing options...
cgchris99 Posted October 11, 2006 Author Share Posted October 11, 2006 How do I get curl to execute the javascript submit on the form?This is the code....alt="Member Login" onclick="doSubmit('Login_loginbutton')" Quote Link to comment Share on other sites More sharing options...
michaellunsford Posted October 11, 2006 Share Posted October 11, 2006 You have to read the javascript and see what it does. More than likely, it just makes sure the form has been filled in properly. Just make sure you're posting it to the same place "action" goes to (if there isn't an action, just post it to the starting page). Quote Link to comment Share on other sites More sharing options...
cgchris99 Posted October 11, 2006 Author Share Posted October 11, 2006 thanks. I figured out what the post did via the javascript. I think it is working fine now. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.