fopen with a .htaccess popup authentication

[RSR]

I'm in the middle of a school project in a programming class and i ran into some problems.

 

I'm using the following code to access another site and read the source code, which i use to post some other information on my own site.

 

$url = "http://othersite.com/";
$fd = fopen($url, 'r');
while (!feof ($fd)){
    $buffer = fgets($fd, 4096);
    $lines[] = $buffer;
}
fclose ($fd);

 

This code worked for me, but then the other site started using a login popup thing (some .htaccess thing, I guess).

 

I can login on the site by typing this in my browser:

http://user:pass@othersite.com/

 

So I thought that i could just add that to my code:

 

$url = "user:pass@http://othersite.com/";
$fd = fopen($url, 'r');
while (!feof ($fd)){
    $buffer = fgets($fd, 4096);
    $lines[] = $buffer;
}
fclose ($fd);

 

This dosn't work for some reason?

So my question is: How to read the source code of the site, when they are using a .htaccess login?

[RSR]

I got this code from an example on php.net:

$c = curl_init();
curl_setopt($c, CURLOPT_URL, "http://othersite.com/");
curl_setopt($c, CURLOPT_USERPWD, "user:pass"); 
curl_exec($c);
curl_close($c);

This code gets the source code of the site, but it dosn't use the username and password correctly, so I'm getting an error: "401 - Unauthorized: Access is denied due to invalid credentials."

[thebadbad]

You wrote user:pass@http://othersite.com/ instead of http://user:pass@othersite.com/. I just tested the method on a folder protected with a .htaccess username and password, and it worked.

[RSR]

Oh, well I did type 'http://user:pass@othersite.com' in the real code.

 

This is the site I'm trying to access: htx.ots.dk/otgnet/adm/restr/portfolio/listeafleveringer.asp

 

I get this error:

Warning: fopen(http://...@htx.ots.dk/otgnet/adm/restr/portfolio/listeafleveringer.asp) [function.fopen]: failed to open stream: HTTP request failed! HTTP/1.1 401 Unauthorized in mysite\....\funktioner.php on line 717

 

With this code:

716: $url = "http://user:pass@htx.ots.dk/otgnet/adm/restr/portfolio/listeafleveringer.asp";
717: $fd = fopen($url, 'r');
718: while (!feof ($fd)){
719:     $buffer = fgets($fd, 4096);
720:     $lines[] = $buffer;
721: }
722: fclose ($fd);

[thebadbad]

Well, I don't know why it doesn't work. It did for me when testing on another site. Just double check your username and password.

[RSR]

Hmm, i just found out that it is only google chrome that accepts the url:

http://user:pass@htx.ots.dk/otgnet/adm/restr/portfolio/listeafleveringer.asp

 

Firefox pops up with the login form even though the username and password is in the url.

 

IE gives me an error:

Windows cannot find 'http://user:pass@htx.ots.dk/otgnet/adm/restr/portfolio/listeafleveringer.asp'. Check the spelling and try again.

 

 

I'm starting to think that it might not be a .htaccess login?

 

Could it be some asp thing?

 

 

EDIT:

 

A thing just crossed my mind. The username is '***\v-******' where the * are letters and numbers.

When the url works in google chrome, I only type v-******:

http://v-******:pass@htx.ots.dk/otgnet/adm/restr/portfolio/listeafleveringer.asp

 

I have tried both ***\v-****** and v-****** in the php code, but neither of them worked.

 

I'm thinking that it might be something with the '-' in the code that is causing some problem?

[thebadbad]

Oh, yeah, must be a Microsoft thing. But I've got no idea how to deal with that.

 

Edit: If the username or password contains backslashes, you should enclose the whole URL in single quotes instead of double quotes to avoid any character translation. Try that and see if it does anything.

[RSR]

Hmm, I found out that the '***\' part isn't necessary.

And the changing of " to ' didn't work.

 

I guess it's not a .htaccess login then?

Do you know any other way you could read the source code from a site?

 

I tried some cURL example, this displayed the source code of the servers no-access page.

So this couldn't login either.

 

    $url = 'htx.ots.dk/otgnet/adm/restr/portfolio/listeafleveringer.asp';

    $ch = curl_init();

    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_USERPWD, "user:pass");   
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

    $output = curl_exec($ch);

    curl_close($ch);
    
    print $output;

[thebadbad]

must be a Microsoft thing. But I've got no idea how to deal with that.

 

Edit: Found an article that explains the backslash thingy though: http://msdn.microsoft.com/en-us/library/72wdk8cc(vs.71).aspx

But still don't know how to login via the URL.

[RSR]

Oh, you meant that the login was a microsoft thing and you don't know how to do that. Now I get it.

But thanks for your time anyways (:

[RSR]

Edit: Found an article that explains the backslash thingy though: http://msdn.microsoft.com/en-us/library/72wdk8cc(vs.71).aspx

But still don't know how to login via the URL.

 

Oh, so the '***\' is just the domain. Hmm, it is strange that it is possible to login with or without the domain.

[mattal999]

Well, to my knowledge, the http://user:pass@domain.com/ isn't a used thing anymore. It is more known for people to just type in their username and password on a login popup.

 

Ok. Well since I don't have cURL installed on my local server (PHP4), I can't do any tests. Can you send me the username and password over a PM so I can test the script on my server?

[thebadbad]

Well, to my knowledge, the http://user:pass@domain.com/ isn't a used thing anymore. It is more known for people to just type in their username and password on a login popup.

 

While I agree that protection of files via Apache's or maybe ASP.NET's authorization feature is rarely used these days (and accessing the files via the userinfo URI component even less), it is irrelevant to the case. He needs a way to automate the login via a script.

 

@OP

If you're studying on that school the site belongs to, I would ask an IT guy if it's possible to access the protected file via PHP somehow. Maybe FTP would be a possibility, if it's impossible the other way around.

[RSR]

Ok. Well since I don't have cURL installed on my local server (PHP4), I can't do any tests. Can you send me the username and password over a PM so I can test the script on my server?

 

I'm afraid that I can't send you my username and password. But if you could send me your test ideas, I would be happy to test it myself.

 

 

@OP

If you're studying on that school the site belongs to, I would ask an IT guy if it's possible to access the protected file via PHP somehow. Maybe FTP would be a possibility, if it's impossible the other way around.

 

Your right, I can ask my teacher if he knows any way to do it. But the problem is that the site is actually not owned by my school, but a larger school in another city. My school are only using it.

[thebadbad]

I just by chance stumbled upon an interesting cURL setting (CURLOPT_HTTPAUTH) that decides which authentication method to use with the connection. Have no idea why I didn't see it earlier. Try this, I actually think it should work:

 

<?php
$url = 'http://htx.ots.dk/otgnet/adm/restr/portfolio/listeafleveringer.asp';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_USERPWD, 'username:password');
curl_setopt($ch, CURLOPT_FORBID_REUSE, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 6.0; da; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11');
$contents = curl_exec($ch);
if ($contents === false) {
trigger_error('Failed to execute cURL session: ' . curl_error($ch), E_USER_ERROR);
}
echo $contents;
?>

[RSR]

I just by chance stumbled upon an interesting cURL setting (CURLOPT_HTTPAUTH) that decides which authentication method to use with the connection. Have no idea why I didn't see it earlier. Try this, I actually think it should work:

 

<?php
$url = 'http://htx.ots.dk/otgnet/adm/restr/portfolio/listeafleveringer.asp';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_USERPWD, 'username:password');
curl_setopt($ch, CURLOPT_FORBID_REUSE, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 6.0; da; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11');
$contents = curl_exec($ch);
if ($contents === false) {
trigger_error('Failed to execute cURL session: ' . curl_error($ch), E_USER_ERROR);
}
echo $contents;
?>

 

It works!

I actually have seen this htttpauth thing before, but I didn't know how to use it, so I forgot about it.

 

Thank you alot for your time and help.

I just love when some tricky little problem gets solved.

Thank you (:

[thebadbad]

Glad it works, and you're welcome