Jump to content


Photo

Crawl password-protected website


  • Please log in to reply
2 replies to this topic

#1 oyoyg

oyoyg
  • New Members
  • Pip
  • Newbie
  • 2 posts

Posted 28 April 2006 - 06:32 PM

Hi

How can I crawl a password-protected site? (I have a password, and its not illegal because its my own site)

#2 ober

ober
  • Staff Alumni
  • Advanced Member
  • 5,337 posts
  • LocationEast Coast, USA

Posted 28 April 2006 - 07:31 PM

You're going to have to explain yourself a little better. What are you trying to achieve? And if your crawler is smart enough to submit forms, you can certainly crawl whatever is past the authentication.

Info: PHP Manual


#3 oyoyg

oyoyg
  • New Members
  • Pip
  • Newbie
  • 2 posts

Posted 28 April 2006 - 10:06 PM

[!--quoteo(post=369675:date=Apr 28 2006, 02:31 PM:name=ober)--][div class=\'quotetop\']QUOTE(ober @ Apr 28 2006, 02:31 PM) View Post[/div][div class=\'quotemain\'][!--quotec--]
You're going to have to explain yourself a little better. What are you trying to achieve? And if your crawler is smart enough to submit forms, you can certainly crawl whatever is past the authentication.
[/quote]

Yeah, I wan't to give some login information to my crawler, and it has to use it on a login form. After that the session should be set, and it should be possible to crawl the pass protected sites, right?




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users