Using CURL to submit form via post method - Tutorial?

[doni49]

Does anyone out there know of a GOOD tutorial for using CURL in PHP?  Printf provided me with some code that I've used to grab a page's contents.  I ran the contents through a regex and successfully got all of the input fields for the first page.  I need to enter the letter "E" in one of those fields and simulate the submit button.  There are THREE seperate buttons on the page of type SUBMIT.  I need to simulate the one named "Transmit". 

All the CURL tutorials that I've been able to find are either for OTHER languages or perform specific tasks and I wasn't able to figure out how to translate them to my task.

The following are the input fields that were found on the first page.

[code]
            <input name="action" type="hidden" value="transmit">
            <INPUT NAME=Fld1 TYPE=Text SIZE=2 MAXLENGTH=2  VALUE="">
            <INPUT NAME=Fld2 TYPE=Text SIZE=1 MAXLENGTH=1  VALUE="">
            <input type="submit" style="width:75px" name="transmit" value="Transmit">
            <input type="submit" style="width:75px" name="quit" value="Quit">              <--if I click on this, I'm sent to a page OUTSIDE the application
            <input type="submit" style="width:75px" name="exit" value="Exit">              <--if I click on this, I'm sent to a page OUTSIDE the application
                                                                                              but different than the one above.
[/code]

[btherl]

The post syntax for variables is the same as get syntax, it just goes in a different location.. for example:

[code=php:0]curl_setopt($ch, CURLOPT_POST, 1);
$post_args = "action=transmit&Fld1=foo&Fld2=bar&submit=transmit";
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_args);
$content = curl_exec($ch);[/code]


That's all there is to using post with curl, providing it's only variables you want to post.  $ch will need to be initialized already, which will be the same as in the code you have.  And whether you get the content as the return value from curl_exec() depends on the CURLOPT_RETURN_TRANSFER flag

[doni49]

Oh--ok so the "submit=" part is what simulates pressing the button?  If it's "value" were something else, I'd just type THAT value in?

If so, that makes sense.  Also, I've been reading more about this, and I thought I saw that I had to use urlencode on the value portions of the fields.  Are you saying that I can just pass "straight text"?

Thanks!

[btherl]

yep, that's all there is to simulating a button press (providing there's no javascript).  A submit button has a name and a value, and those are passed as variables.  A button that isn't pressed doesn't have its name and value passed.

Hmm.. I think you would have to urlencode, otherwise embedded '=' and '&' characters would mess up your variables.  Not to mention newlines and other things in your variable values.

[doni49]

I'm getting an error message:
[quote]
Warning: curl_setopt(): 1 is not a valid cURL handle resource in /home2/donirela/public_html/beta/getdata3.php on line 24
[/quote]

I've tried BOTH of the following lines on line 24:
[code]
curl_setopt ( $io, CURLOPT_POSTFIELDS, $postfields)
[/code]
and
[code]
curl_setopt ( $io, CURLOPT_POSTFIELDS, urlencode($postfields));
[/code]

And this is what $postfields looks like:
[quote]
action=&Fld1=&Fld2=&transmit=Transmit
[/quote]

[btherl]

Have you initialized the curl resource, using something like

[code=php:0]$io = curl_init('http://www.domain.com/script.php');[/code]


The error is complaining about $io, not about the other arguments.

[doni49]

I was just trying to post this, when your message came in.

P.S. $io is initialized to use CURL--this is actually the second time through.  The first time worked fine.

I retrieved the first page, parsed out the input fields and then attempt to post them.  This is the attempt to post them.

[btherl]

Can you post your entire code?  Starting from the curl_init() at least.

[doni49]

[code]
<?php
$tst = 1;
while ($tst > 0){
      switch ($tst){
        case 1:
            $url ='http://4.43.65.248/autoform.asp?app=cvr';
            break;
        case 2:
        $page = "";
            $url ='http://4.43.65.248/autoform.asp';
            //preg_match('<form method="post".*action="(.*)[\s]+.*">',$page,$method);
            for($i=0;$i<count($matches);$i++){
                for($k=0;$k<count($matches[$i]);$k++){
                  $matches[$i][$k]=str_replace('"','',$matches[$i][$k]);
                  $matches[$i][$k]=str_replace("'","",$matches[$i][$k]);
                }
            }
            $postfields = $matches[0][1] . "=" . $matches[$i][2];
            for($i=1;$i<count($matches);$i++){
              $postfields .= "&" . $matches[$i][1] . "=" . urlencode($matches[$i][2]);
            }
            //echo $postfields;
            curl_setopt ( $io, CURLOPT_POSTFIELDS, $postfields);//urlencode($postfields));
            break;
        case 3:
            break;
      }


      $ref = 'http://4.43.65.248/autoform.asp';
      $jar = '/curl-cookies/cookie.txt';

      $io = curl_init ();

      // the url to fetch, string (uri)
      curl_setopt ( $io, CURLOPT_URL, $url );

      // ? return the response header (0 = no, 1 yes)
      curl_setopt ( $io, CURLOPT_HEADER, 0 );

      // set the referer
      curl_setopt ($io, CURLOPT_REFERER, $ref);

      // ? the connection time out (int)
    curl_setopt ( $io, CURLOPT_TIMEOUT, 5 );

      // ? expect [chunck, gzip] transfers (empty str)
      curl_setopt ( $io, CURLOPT_ENCODING, '' );

      // ? maximum redirect(s) to follow (int)
      curl_setopt ( $io, CURLOPT_MAXREDIRS, 1 );

      // ? where to save [Set-Cookie] (str, path + file name)
      curl_setopt ( $io, CURLOPT_COOKIEJAR, $jar );

      // ? where to read and send [Cookie] (str, path + file name)
      curl_setopt ( $io, CURLOPT_COOKIEFILE, $jar );

      // ? follow location headers (0 = no, 1 = yes)
      curl_setopt ( $io, CURLOPT_FOLLOWLOCATION, 1 );

      // ? return data to curl_exec(), (0 = no, 1 = yes)
      curl_setopt ( $io, CURLOPT_RETURNTRANSFER, 1 );

      // the user agent
      curl_setopt ( $io, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)' );

      // execute the request
      $page = trim ( curl_exec ( $io ) );

      // release the resource
      curl_close ( $io );


      switch ($tst){
        case 1:
            $page = preg_replace('/<INPUT.*?NAME="quit".*?>/imU','',$page);
            $page = preg_replace('/<INPUT.*?NAME="exit".*?>/imU','',$page);
            preg_match_all('/<INPUT.*?NAME=(.*)[\s]+.*VALUE="(.*)">/imU',$page,$matches, PREG_SET_ORDER);
            $tst++;
            break;
        case 2:
            echo $page;
            $tst++;
            break;
        case 3:
            $tst = false;
            break;
      }
}
?>
[/code]

[doni49]

I set $tst to 1 then I start a while loop so that until $tst is less than 1, it keeps moving to the next step.

Step 1 is to load the page and get all the fields.  Then move on to step 2 which is to POST the fields.  The first set of switch code performs the task of deciding which step do.  Then the curl_opts are the same (except the fields) for all the attempts so it's after the switch is over.

Then after that is done, parse the data in the second set of switch code.

[doni49]

I [b]think[/b] maybe I see it.  Maybe I need to put the init statement ABOVE the switch statement.  I'll try that tomorrow night.

I added the switch statements after I had it getting the first page.  I was thinking that I had it successfully getting the first page AFTER I added the switch statements.  But now I'm not sure about that.

[btherl]

Yes, I think that's it.  You can't set the curl options before initializing the object.  I think it should be fine to set the post fields before setting the url.. not 100% sure on that.

Your url encoding for the post fields looks good :)

[doni49]

Thanks for the help.  That DID get rid of the error.  But it's still not displaying the second page.  I'll have to get back to this when I get back from my Thanksgiving trip.  I'll likely be back asking more questions then.

Do you know of a good tutorial out there?

[hitman6003]

If you want to post to another page a much easier way read this page:

http://us2.php.net/manual/en/wrappers.http.php#AEN271674

It uses only three functions, and none of them have to do with the much more difficult cURL.

[doni49]

I don't know if the problem is with the posting of the fields, something with the referer or who knows what else.

When visiting the page manually, you go to the first page and are prompted to enter a letter into the text field (the first page is basically a menu to choose what type of records you need).  Then when you press "Transmit" (a submit button labeled "Transmit"), it sends you to the next page (which is actually the same web page with different variables/values in the URL).

Any ideas how I would determine whether the error(s) is in the fields, referer or otherwise?

Thanks.

[btherl]

There are some firefox extensions, including UrlParams, Tamper Data and Live HTTP Headers, which will let you log all the headers and post data from an actual request.  You can compare that with your script, and add anything you think might make a difference.  Usually it's the referrer, the user-agent or some hidden post field which is missing.  Some sites need a "jsessionid" in the url.

Unfortunately it's just trial and error.. They're not about to give you an API for the site :)