Question on how to mine form data

[jeffjohnvol]

Hello, its been a few years since I've been here, I like the new look, very clean looking.

 

Anyway, I have a task assigment for a property owners association.  I know how to navigate to get this data, but I don't know how to programatically grab it.

 

Manually, you would go to http://www.assessment.state.tn.us/SelectCounty.asp , put CARTER for the county, HORSESHOE in the subdivision name, and hit SEARCH.

 

This will produce a list of properties.  I need to navigate to each one (select radio button) to get to the page data.

 

The page data is such that if I could get a string with the information, I could parse it out easily. 

 

THe data gets refreshed rarely, so I would only need to run this once a month or so.  I wouldn't expect it to happen anytime a page was executed, as I know it would take too long.

 

Any suggestions, gang?

[PFMaBiSmAd]

There are a ton of existing site scraper scripts, though you would need to modify any of them for the specific site.

 

Forms only really exist after they have been rendered in a browser. For a script to get data from a site, you actually submit data directly to the form processing page. The easiest method is to use the curl extension.

 

You would submit data to the first form processing page and capture the response, parse through the results to pick out what you submit for each property, then submit those values to the correct form processing page and capture the responses.

[.josh]

You can use cURL to post form data and the results page will be returned as a string of the actual html content (like when you rightclick > viewsource).  From there you can use DOM to parse the page for info or use your own custom regex.

[jeffjohnvol]

I'll do some googling on cURL and DOM to find examples that I can work with.  Thanks for pointing me in the right direction.

 

Jeff