Jump to content

Archived

This topic is now archived and is closed to further replies.

oyoyg

Crawl password-protected website

Recommended Posts

Hi

How can I crawl a password-protected site? (I have a password, and its not illegal because its my own site)

Share this post


Link to post
Share on other sites
You're going to have to explain yourself a little better. What are you trying to achieve? And if your crawler is smart enough to submit forms, you can certainly crawl whatever is past the authentication.

Share this post


Link to post
Share on other sites
[!--quoteo(post=369675:date=Apr 28 2006, 02:31 PM:name=ober)--][div class=\'quotetop\']QUOTE(ober @ Apr 28 2006, 02:31 PM) [snapback]369675[/snapback][/div][div class=\'quotemain\'][!--quotec--]
You're going to have to explain yourself a little better. What are you trying to achieve? And if your crawler is smart enough to submit forms, you can certainly crawl whatever is past the authentication.
[/quote]

Yeah, I wan't to give some login information to my crawler, and it has to use it on a login form. After that the session should be set, and it should be possible to crawl the pass protected sites, right?

Share this post


Link to post
Share on other sites

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.