ultimatum Posted September 25, 2012 Share Posted September 25, 2012 Hey guys, I have a page that I need to scrape using PHP. This page requires login credentials (which I have) but I don't know how to pass those values to a login prompt (pop-up). It's a simple prompt like this one: http://devot-ee.com/...ogin_prompt.png Trying to just read the page using cURL or file_get_contents_page() function sends back "You are not authorized to see this page" without login questions or anything related. If anyone knows of a solution please help me out. Thank you, - ultimatum Quote Link to comment https://forums.phpfreaks.com/topic/268765-how-to-scrape-a-page-that-uses-prompt-login/ Share on other sites More sharing options...
requinix Posted September 25, 2012 Share Posted September 25, 2012 You're logging into your own site? That's HTTP authentication. Use cURL and curl_setopt with the right option - I forget what it's called but you'll know it when you see it. Quote Link to comment https://forums.phpfreaks.com/topic/268765-how-to-scrape-a-page-that-uses-prompt-login/#findComment-1380743 Share on other sites More sharing options...
ultimatum Posted September 26, 2012 Author Share Posted September 26, 2012 No, I'm not. If it was my site then I would just connect to the database. I'm parsing a website which I have a subscription to and parsing has been allowed to automate some steps in data analysis. Are you talking about this CURLOPT_HTTPAUTH option? If I get through the login, will curl pick up webpages with frames and will I be able to scrape information from them? Thank you. Quote Link to comment https://forums.phpfreaks.com/topic/268765-how-to-scrape-a-page-that-uses-prompt-login/#findComment-1381100 Share on other sites More sharing options...
requinix Posted September 26, 2012 Share Posted September 26, 2012 (edited) CURLOPT_USERPWD Frames are dealt with by the browser - cURL will only give you the HTML that defines the frame, not the contents of the page referenced. In general you do this kind of work by logging in (and making sure cURL is saving the cookies someplace you can reuse in subsequent calls) and hitting the exact URLs you need. Edited September 26, 2012 by requinix Quote Link to comment https://forums.phpfreaks.com/topic/268765-how-to-scrape-a-page-that-uses-prompt-login/#findComment-1381167 Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.