PHP CURL past a form to second page?

[Rifts]

Hey everyone,

 

I'm a little new to using CURL and slightly confused. I'm using CURL to access my URL with a form. I'm trying to fill and submit the form, then scrape the result from the loaded page.

 

Is this even possible? How do I follow the form submission?

 

thanks in advanced!

[.josh]

Yes it's possible. Basically you take the form's action url and use that for your curl request. If the form is submitted with the GET method, you just attach form fields and values like a query string to the url.

 

If the form is submitted via POST method, you need to use curl_setopt to set CURLOPT_POST to true and use CURLOPT_POSTFIELDS to pass the form data.

 

You may also need to set CURLOPT_FOLLOWLOCATION true in case the form processing script performs a redirect, to allow for the curl request to follow it.

 

You will also need to set CURLOPT_RETURNTRANSFER to capture the response of the curl request.

 

 

If the form requires SSL connection, you will need to set additional parameters.

 

If the form processing script looks at other things like user agent or cookies to determine if it's a bot trying to spam or other things required by the site, you may need to add additional settings to make it look more like a browser made the request.

 

Then you will use curl_exec to perform the curl request and return the results from the server.

 

Methods for scraping the results largely depends on what kind of results are returned and what you are trying to scrape. You may need to use the DOM parser you may need to use some regex. Orr..the site might try to be tricky and render stuff with javascript to deter scraping, in which case it will be really hard to get what you want, short of running the results through an html/js parser (like phantomjs).

[Rifts]

Thank you for this amazing and complete response. This is exactly what I was looking for!