How To Scrape A Page That Uses Prompt Login

[ultimatum]

Hey guys,

 

I have a page that I need to scrape using PHP. This page requires login credentials (which I have) but I don't know how to pass those values to a login prompt (pop-up).

 

It's a simple prompt like this one: http://devot-ee.com/...ogin_prompt.png

 

Trying to just read the page using cURL or file_get_contents_page() function sends back "You are not authorized to see this page" without login questions or anything related.

 

If anyone knows of a solution please help me out.

 

Thank you,

 

 

- ultimatum

[requinix]

You're logging into your own site?

 

That's HTTP authentication. Use cURL and curl_setopt with the right option - I forget what it's called but you'll know it when you see it.

[ultimatum]

No, I'm not. If it was my site then I would just connect to the database. I'm parsing a website which I have a subscription to and parsing has been allowed to automate some steps in data analysis.

 

Are you talking about this CURLOPT_HTTPAUTH option?

 

If I get through the login, will curl pick up webpages with frames and will I be able to scrape information from them?

 

 

Thank you.

[requinix]

CURLOPT_USERPWD

 

Frames are dealt with by the browser - cURL will only give you the HTML that defines the frame, not the contents of the page referenced.

 

In general you do this kind of work by logging in (and making sure cURL is saving the cookies someplace you can reuse in subsequent calls) and hitting the exact URLs you need.

Edited September 26, 2012 by requinix