file get contents with complicated (?) login first

[muppet77]

i am trying to do a "file get contents" from the following website
http://www.bet365.com/home/

It is necessary to input your account username and password here, and then I want to get the contents of this page
http://www.bet365.com/extra/en/results-and-archive/results/?Fromdate=13%2f09%2f2014+00%3a00&SearchPath=result&FixtureId=46827110&Period=99&ChallengeId=26278650&CompetitionId=999999&SportId=1&SportDesc=Soccer&Todate=14%2f09%2f2014+00%3a00&LanguageId=1&Zoneid=1

I have tried a couple of scripts but just can't seem to get it to work.

I have tried


$context = stream_context_create(array(
    'http' => array(
        'header'  => "Authorization: Basic " . base64_encode("$username:$password")
    )
));
$data = file_get_contents($url, false, $context);

echo $data;


with the username and password and url entered as variables.
but get the message

Warning: file_get_contents(http://www.bet365.com/extra/en/results-and-archive/results/?Fromdate=13%2f09%2f2014+00%3a00&SearchPath=result&FixtureId=46827110&Period=99&ChallengeId=26278650&CompetitionId=999999&SportId=1&SportDesc=Soccer&Todate=14%2f09%2f2014+00%3a00&LanguageId=1&Zoneid=1) [function.file-get-contents]: failed to open stream: HTTP request failed! in "myscript's address"  on line 13


Maybe i don't need to visit the homepage first to log in?
Any ideas please?
Thank you

apologies for the message being in code quotes, I wrote it in Word but then couldn't paste it, even using the special Word paste button.

Hope the message comes out ok.

Thank you.

 

[Jacques1]

This site uses a cookie-based session system. If you inspect the cookies in your browser, you'll see one called “ASP.NET_SessionId”. This obviously carries the session ID.

 

So you need to send your credentials to their ASP.NET application, store the session cookie and then send it along with every request. This is way beyond the capablities of file_get_contents(). Use the cURL library instead.

 

Note that you'll need a basic understanding of how HTTP works.

[muppet77]

thank you.

please could you direct me to a good starting point for this - something similar?

 

do you think  that it is possible?

 

I am a novice but keen to learn. don't know much about curls but a little about php other than this.

[Jacques1]

There are literally hundreds of examples on Google. Why don't you look for them yourself and come back when you have a concrete question?

 

To get familar with the authentication procedure, it might be a good idea to set up a simple test application on your localhost. Make one log-in script and one script which requires a valid session. See if you can post your credentials to the log-in script and store the session cookie you get back. If that works, see if you can access the protected script with the stored cookie.

[muppet77]

Thank you I will do.

 

So if I search for

Curl log in

??

 

Please could you advise me on what I am looking for ? Really basic question but is it just curl?

Edited September 18, 2014 by muppet77

[Jacques1]

Search for something like “php curl login”. On the first page, there's already a Stack Overflow thread where this problem is discussed in great detail and with many examples.

 

The relevant cURL options are CURLOPT_POST and CURLOPT_POSTFIELDS for making the initial POST request. You'll also need CURLOPT_COOKIEJAR for writing the session cookie to a file and CURLOPT_COOKIEFILE for reading the cookie from that file.

[muppet77]

ok, had a look at those, thank you.

I get a blank page returned from this code:

//init curl
$ch = curl_init();

//Set the URL to work with
curl_setopt($ch, CURLOPT_URL, $url);

// ENABLE HTTP POST
curl_setopt($ch, CURLOPT_POST, 1);

//Set the post parameters
curl_setopt($ch, CURLOPT_POSTFIELDS, 'user='.$username.'&pass='.$password);

//Handle cookies for the login
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');

//Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL
//not to print out the results of its query.
//Instead, it will return the results as a string return value
//from curl_exec() instead of the usual true/false.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

//execute the request (the login)
$store = curl_exec($ch);

//the login is now done and you can continue to get the
//protected content.

//set the URL to the protected file
curl_setopt($ch, CURLOPT_URL, $grab);

//execute the request
$content = curl_exec($ch);

curl_close($ch);


echo $content;

$username, $password are set as my login variables

and

$url = 'http://www.bet365.com';
$grab = 'http://www.bet365.com/extra/en/results-and-archive/results/?Fromdate=13%2f09%2f2014+00%3a00&SearchPath=result&FixtureId=46827110&Period=99&ChallengeId=26278650&CompetitionId=999999&SportId=1&SportDesc=Soccer&Todate=14%2f09%2f2014+00%3a00&LanguageId=1&Zoneid=1';
 

any suggestions please?

[Jacques1]

You're not sending the cookies anywhere. And the entire log-in procedure looked more complicated to me than just sending the name and password to the site.

 

Like I said, do this step by step. Create a simple test script on your PC and try out cURL in a controlled environment before you jump to some live site.

[muppet77]

Thanks Jacques1, I appreciate your advice.

I am afraid I have no idea what a simple test script would look like?

I am well in the deep end here.

[muppet77]

i know that if i go to this page

 

http://www.bet365.com/extra/en/results-and-archive/results/?fromdate=13/09/2014%2000:00&searchpath=result&fixtureid=46827110&period=99&challengeid=26278650&competitionid=999999&sportid=1&sportdesc=soccer&todate=14/09/2014%2000:00&languageid=1&zoneid=1

 

i then log in and it arrives back at the same page, but with full access. There is no other page to visit, as I thought before.

[jazzman1]

This login page  requires javascript to be enabled in your browser. No way to use cURL.

Edited September 19, 2014 by jazzman1

[Jacques1]

I am afraid I have no idea what a simple test script would look like?

 

Like I explained in #4: One script for logging in and one script which checks the log-in status. The log-in script can be as simple as

<?php

session_start();

if ($_POST['name'] == 'foo' && $_POST['password'] == 'bar')
{
	$_SESSION['logged_in'] = true;
	echo 'Log-in successful';
}
else
{
	echo 'Log-in failed';
}

The point is that you should test and fix your cURL code before you go to the next step of using it on an actual site. It's not helpful to end up with some blank page and no idea what might be wrong.

 

 

 

 

 

 

This login page  requires javascript to be enabled in your browser. No way to use cURL.

 

Um, what? You do realize that the end result is always an HTTP request, right? Or did you think JavaScript performs some kind of magic?

Edited September 19, 2014 by Jacques1

[jazzman1]

Ok, how will you send an ajax request (if the script is required) without javascript?

[CroNiX]

You send the request to the script that ajax would normally be submitting the login info to. It doesn't need to be an "ajax" request, unless they are specifically looking for the XMLHttpRequest header in which case you'd just need to add it to the CURL header to simulate an ajax request.

Something like:

curl_setopt($ch, CURLOPT_HTTPHEADER, array("X-Requested-With: XMLHttpRequest", "Content-Type: application/json; charset=utf-8"));

Edited September 19, 2014 by CroNiX

[Jacques1]

Ok, how will you send an ajax request (if the script is required) without javascript?

 

An Ajax request is an HTTP request. Have you never opened the “network” tab of your browser to see what's going on behind the scenes?

 

It doesn't matter if the request comes from JavaScript or cURL or whatever. You might as well write it down by hand. The server doesn't know and doesn't care. It's all just a bunch of bytes in a bunch of IP packets.

Edited September 19, 2014 by Jacques1

[muppet77]

Like I explained in #4: One script for logging in and one script which checks the log-in status. The log-in script can be as simple as

 

<?php

session_start();

if ($_POST['name'] == 'foo' && $_POST['password'] == 'bar')
{
	$_SESSION['logged_in'] = true;
	echo 'Log-in successful';
}
else
{
	echo 'Log-in failed';
}

The point is that you should test and fix your cURL code before you go to the next step of using it on an actual site. It's not helpful to end up with some blank page and no idea what might be wrong.

 

 

 

 

 

 

 

 

 

 

Um, what? You do realize that the end result is always an HTTP request, right? Or did you think JavaScript performs some kind of magic?

Thanks. Unfortunately my knowledge is not that good.

I don't understand how that script logs in anywhere.

Really sorry!

[jazzman1]

Jacques, who is talking about how the server treats the http post/get form? I am talking about that there is no way to make an ajax call without javascript and this could be easily checked (if js is enabled/disabled by php ( php is cross-side language))

 

CroNiX in the official docs of cURL, there is nothing about js and curl implementation. I don't think your example is possible to send data in this way by curl.

 

muppet77, what is your username and password. Send me a PM if you want it, to check if it's possible to use curl and lately I'll post the solution to the forum.         

Edited September 19, 2014 by jazzman1

[muppet77]

Hi jazzman. I am not keen to share my user details (even via DM). Really sorry but hope you understand.

Could you (or me?) regester another account? It's free.

 

Thanks for your time.

[jazzman1]

 

Could you (or me?) regester another account? It's free.

 

Ok, do it for me and send me the login credentials.

[muppet77]

DM ing you now JM

[Jacques1]

No offense, jazzman, but I think you're very confused.

 

The communication between the client and the server consists of exchanging IP packets (which contain the HTTP messages). The server doesn't know anything about the client except the IP address. It has absolutely no idea what the client did to generate the data: Could be Ajax, could be a “classical” request, could be cURL or any other tool. Hell, the client could have manually typed in the bytes with their keyboard. We don't know, and we cannot find out. All we know is that a particular IP address has sent us an HTTP request.

 

For the server, there's no such thing as “Ajax”. This is a client-side technique, and you have no way to detect or enforce it.

 

Think of the client as a black box: Data goes in, data goes out, but what happens inside the box is completely unknown. Is it a human user or a bot? Do they use a browser? If they use a browser, do they have JavaScript enabled? We have no idea.

Edited September 19, 2014 by Jacques1

[jazzman1]

You are still continue to teach me how the server and client works. No problem, no offense is taken by me Now, I already know that there is not way php to detect if javascript is enabled / disabled in my browser when I make an ajax call to the server. So, back to the OP problem, here is the logic credentials:

#!/bin/bash

NAME='user'
PASS='pass'
referer='http%3a%2f%2fwww.bet365.com%2fxtra%2fen%2f'

#curl --cookie-jar cjar --output /dev/null 'https://members.bet365.com/Members/lp/default.aspx'

curl --cookie cjar --cookie-jar cjar 'https://members.bet365.com/Members/lp/default.aspx' --data "ctl00_AjaxScriptManager_HiddenField=&__EVENTTARGET=ctl00%24HeaderPlaceHolder%24HeaderControl%24Login%24Go&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwULLTE0OTk1NjUxNDBkZM%2F826rAqHRdNx1mZCnaiwnKJMB7&__PREVIOUSPAGE=b_rpb_GyZeP1TGPyzN2vQPsEv-IJbnLMMZcHunMJW9JgkSoJrqm6glz93dN4lmrcFcwnZGiTdwRvEiEWZLRm3YVm7wU1&__EVENTVALIDATION=%2FwEWDQKgifDsDwLBpdiBBQKF1rTBCwKg14akCAL%2FnP6QCgK6iKG3CgLx%2BNLkCwLk5KvOAwLBnamGCALUi53qCALxrtCaBwLNvuHaCQK24aRbEcr2XoX5okTIDQsKsnM8zQd6ypk%3D&ctl00%24HeaderPlaceHolder%24ctl00%24hidSessionTimestamp=&ctl00%24HeaderPlaceHolder%24ctl00%24hidSessionHashvalue=&ctl00%24HeaderPlaceHolder%24ctl00%24hidSessionTimeoutDuration=&ctl00%24HeaderPlaceHolder%24ctl00%24hidSessionAppPath=&ctl00%24HeaderPlaceHolder%24HeaderControl%24Login%24Username=${NAME}&ctl00%24HeaderPlaceHolder%24HeaderControl%24Login%24InitialPassword=Password&ctl00%24HeaderPlaceHolder%24HeaderControl%24Login%24ProtectedPassword=${PASS}&txtPassword=${PASS}&txtUserName=${NAME}&txtTKN=7EDA0805C01A4AD3AA96165DA59A4461000003&txtLH=0f740fa6750303c821740356981f2e1123e37676&txtTS=20140919204322079&txtType=2&txtSTKN=695060f20f99447ea7406c3d37a75491&txtScreenSize=1280+x+800&txtFlashVersion=11.2.202" --referer=${referer}

curl --cookie cjar --output ~/bet365.html 'http://www.bet365.com/extra/en/'

I'm not finding myself in login action using curl. Ideas instead of teaching each others what is AJAX? 

Edited September 19, 2014 by jazzman1

[muppet77]

Please may you remove the password and user before it's changed!

[Jacques1]

Now, I already know that there is not way php to detect if javascript is enabled / disabled in my browser when I make an ajax call to the server. 

 

Then I have no idea why you told us that it's impossible to make “Ajax requests” with cURL. Was it just a brainfart?

[muppet77]

Jazzman! Please get rid of the log in details!