Please help me with CURL.

[fireflywins]

Hi,

 

I am trying to automated a boring, regular job that I have to do daily. I am trying to automated this job 100% at server end in php by using CURL:

1. Login to a site https://www.tutorsteach.com/login.aspx automatically.

2. then go to this page: https://www.tutorsteach.com/inquiry.aspx.

3. Get the html source code of this page in variable and then scan it for new email ids.

4. Then send them a mail.

 

 

This is my code:

 

<?php

// INIT CURL

$ch = curl_init();

 

// SET URL FOR THE POST FORM LOGIN

curl_setopt($ch, CURLOPT_URL, 'https://www.tutorsteach.com/Default.aspx');

 

// ENABLE HTTP POST

curl_setopt ($ch, CURLOPT_POST, 1);

 

// SET POST PARAMETERS : FORM VALUES FOR EACH FIELD

$EVENTTARGET = "left1$lnkLogin";

$EVENTARGUMENT = "";

$VIEWSTATE = "/wEPDwUKMTcwOTg5NzA3OA9kFgICAQ9kFgQCBQ9kFgICAQ9kFgICAQ8WAh4EVGV4dAWkKTxwPjxzdHJvbmc+V2VsY29tZSB0byBUdXRvcnNUZWFjaC5jb20gLS0gPGZvbnQgY29sb3I9ImJyb3duIj5UaGUgbGFyZ2VzdCB0dXRvciBwcm92aWRlciBpbiB0aGUgcmVnaW9uITwvZm9udD48L3N0cm9uZz48YnIvPjxpPlNpbmNlIDE5OTk8L2k+PC9wPg0KPHA+PGEgaHJlZj0iRmxhc2gvdG91ci5odG0iIG9uY2xpY2s9IndpbmRvdy5vcGVuKCdGbGFzaC90b3VyLmh0bScsJ1JJQycsJ3dpZHRoPTYwMCxoZWlnaHQ9NDM4LHJlc2l6YWJsZT1ubyxzY3JvbGxiYXJzPW5vLHRvb2xiYXI9bm8sbWVudWJhcj1ubyxsb2NhdGlvbj1ubyxzdGF0dXM9bm8sbGVmdD0xMDAsdG9wPTQwLHNjcmVlblg9MTAwLHNjcmVlblk9NDAnKTtyZXR1cm4gZmFsc2U7Ij48aW1nIHNyYz0iaW1hZ2VzL1RvdXIuanBnIiBhbHQ9IlRha2Ugb3VyIFNUVURFTlQgVE9VUiIgYWxpZ249InJpZ2h0IiBib3JkZXI9IjAiLz48L2E+PC9wPg0KPGRpdiBhbGlnbj0ibGVmdCI+PHN0cm9uZz5UdXRvcnNUZWFjaC5jb208L3N0cm9uZz48YnIvPg0KICAgICZuYnNwOy4uaXMgcHJvdWQgdG8gc3VwcG9ydCB0aGUgPGEgaHJlZj0iQ2hpbGRMZWZ0Lmh0bSIgb25jbGljaz0id2luZG93Lm9wZW4oJ0NoaWxkTGVmdC5odG0nLCdSSUMnLCd3aWR0aD00NTAsaGVpZ2h0PTI2MCxyZXNpemFibGU9bm8sc2Nyb2xsYmFycz15ZXMsdG9vbGJhcj1ubyxtZW51YmFyPW5vLGxvY2F0aW9uPW5vLHN0YXR1cz1ubyxsZWZ0PTE3MCx0b3A9NSxzY3JlZW5YPTE3MCxzY3JlZW5ZPTUnKTtyZXR1cm4gZmFsc2U7Ij5ObyBDaGlsZCBMZWZ0IEJlaGluZDwvYT4gcHJvZ3JhbSENCjwvZGl2Pg0KPHA+PGEgaHJlZj0iaGVsbG8uYXNweCI+PGltZyBzcmM9ImltYWdlcy9PbmxpbmUtVG9vbC5naWYiIGFsdD0iT25saW5lIFR1dG9yaW5nIGFuZCBXaGl0ZWJvYXJkIFRvb2wiIHdpZHRoPSIxMDUiIGFsaWduPSJyaWdodCIgYm9yZGVyPSIwIiBoZWlnaHQ9IjMwIi8+PC9hPjwvcD4NCjxwIGFsaWduPSJjZW50ZXIiPiAuLmFuZCA8c3Ryb25nPjxmb250IGNvbG9yPSJncmVlbiI+Y2VsZWJyYXRlczwvZm9udD4gPGEgaHJlZj0iTW9tRGF5Lmh0bSIgb25DbGljaz0id2luZG93Lm9wZW4oJ01vbURheS5odG0nLCdSSUMnLCd3aWR0aD0zNzUsaGVpZ2h0PTI2MSxyZXNpemFibGU9bm8sc2Nyb2xsYmFycz15ZXMsdG9vbGJhcj1ubyxtZW51YmFyPW5vLGxvY2F0aW9uPW5vLHN0YXR1cz1ubyxsZWZ0PTE1MCx0b3A9MzUsc2NyZWVuWD0xNTAsc2NyZWVuWT0zNScpO3JldHVybiBmYWxzZTsiPjxmb250IGNvbG9yPSJkZWVwcGluayI+PEk+TW90aGVyJ3MgRGF5PC9JPjwvYT48L2ZvbnQ+IDxmb250IGNvbG9yPSJkYXJrZ3JlZW4iPnRoaXMgbW9udGguPC9mb250Pjxici8+PGJyLz48Zm9udCBjb2xvcj0iYnJvd24iPkZSRUUgU2VhcmNoIEZvciBJTi1IT01FIGFuZCBPTkxJTkUgVHV0b3JzPGk+IEFsd2F5czwvaT4gLSBObyBuZWVkIHRvIHNpZ24tdXAhPC9mb250Pjwvc3Ryb25nPjwvcD4NCjx1bD4NCgk8bGk+PGEgaWQ9Imxua19rX2p1bmlvciIgaHJlZj0ibG9jYXRlVHV0b3IuYXNweD9EaXNjaXBsaW5lPTEiPjwvYT48L2xpPg0KCTxsaT48YSBpZD0ibG5rX2hpZ2hzY2hvb2wiIGhyZWY9ImxvY2F0ZVR1dG9yLmFzcHg/RGlzY2lwbGluZT0yIj48L2E+PC9saT4JCQkgICAgDQoJPGxpPjxhIGlkPSJsbmtfY29sbGVnZSIgaHJlZj0ibG9jYXRlVHV0b3IuYXNweD9EaXNjaXBsaW5lPTMiPjwvYT48L2xpPg0KCTxsaT48YSBpZD0ibG5rX3Rlc3RfcHJlcCIgaHJlZj0ibG9jYXRlVHV0b3IuYXNweD9EaXNjaXBsaW5lPTQiPjwvYT48L2xpPg0KCTxsaT48YSBpZD0ibG5rX2ZvcmVpZ25fbGFuZ3VhZ2UiIGhyZWY9ImxvY2F0ZVR1dG9yLmFzcHg/RGlzY2lwbGluZT05Ij48L2E+PC9saT4NCgk8bGk+PGEgaWQ9Imxua19jb21wdXRlciIgaHJlZj0ibG9jYXRlVHV0b3IuYXNweD9EaXNjaXBsaW5lPTgiPjwvYT48L2xpPg0KCTxsaT48YSBpZD0ibG5rX2J1c2luZXNzIiBocmVmPSJsb2NhdGVUdXRvci5hc3B4P0Rpc2NpcGxpbmU9MTAiPjwvYT48L2xpPg0KCTxsaT48YSBpZD0ibG5rX2FydCIgaHJlZj0ibG9jYXRlVHV0b3IuYXNweD9EaXNjaXBsaW5lPTUiPjwvYT48L2xpPg0KCTxsaT48YSBpZD0ibG5rX211c2ljIiBocmVmPSJsb2NhdGVUdXRvci5hc3B4P0Rpc2NpcGxpbmU9NiI+PC9hPjwvbGk+DQoJPGxpPjxhIGlkPSJsbmtfc3BvcnRzIiBocmVmPSJsb2NhdGVUdXRvci5hc3B4P0Rpc2NpcGxpbmU9NyI+PC9hPjwvbGk+DQo8L3VsPg0KPHA+R29vZCB3cml0ZXJzIGtub3cgdGhlcmUgaXMgYSBjb25zaWRlcmFibGUgZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSBhYmlsaXR5IHRvIHB1dCB3b3JkcyBvbiBhIHBhZ2UgYW5kIHRoZSBhYmlsaXR5IHRvIGV4cHJlc3MgeW91cnNlbGYgaW4gbHVjaWQsIGZsb3dpbmcgcGFyYWdyYXBocy4gIENvdW50bGVzcyBzdHVkZW50cyB0aGF0IGNhbiBkbyB0aGUgZm9ybWVyIHN0cnVnZ2xlIHdpdGggdGhlIGxhdHRlciwgYW5kIG9mdGVuIHByYWN0aWNlIGFsb25lIGNhbm5vdCBtYWtlIHRoZSBwcm9ibGVtIGdvIGF3YXkuICBRdWFsaWZpZWQgdHV0b3JzIHdobyBhcmUgZXhwZXJpZW5jZWQgaW4gYWRkcmVzc2luZyBzdWNoIHByb2JsZW1zIGNhbiBtYWtlIGEgY29uc2lkZXJhYmxlIGRpZmZlcmVuY2UuPC9wPjxkaXYgc3R5bGU9ImZsb2F0OnJpZ2h0O2ZvbnQtc2l6ZToxNHB4O2ZvbnQtd2VpZ2h0OiBib2xkO2xpbmUtaGVpZ2h0OjIwcHg7bWFyZ2luOiAwcHggMHB4IDIwcHggMjBweDtwYWRkaW5nOiAxMHB4IDEwcHggMTBweCAxMHB4O3RleHQtYWxpZ246IGNlbnRlcjt3aWR0aDoyMDBweDsiPjxhIHJlbD0ibm9mb2xsb3ciIG9ubW91c2VvdmVyPSJ3aW5kb3cuc3RhdHVzPScnO3JldHVybiB0cnVlOyIgb25tb3VzZW91dD0id2luZG93LnN0YXR1cz0nJztyZXR1cm4gdHJ1ZTsiIGhyZWY9ImxvY2F0ZVR1dG9yLmFzcHgiPkNsaWNrIEhlcmUgZm9yICBXcml0aW5nIFR1dG9ycyE8L2E+PC9kaXY+PHA+V3JpdGluZyBpcyBsaXR0bGUgbW9yZSB0aGFuIHRoZSBwcm9wZXIgb3JnYW5pemF0aW9uIG9mIHlvdXIgdGhvdWdodHMuICBNYW55IG9mIHRoZSBiZXN0IHR1dG9ycyBkZXNpZ24gZXhlcmNpc2VzIHdob3NlIHB1cnBvc2UgaXMgdG8gaGVscCB3aXRoIGdldHRpbmcgY29uY2VwdHMgaW4gdGhlIHByb3BlciBvcmRlciBhbmQgaW1wcm92aW5nIHRoZWlyIGV4ZWN1dGlvbi4gIFN0dWRlbnRzIHdobyBzdHJ1Z2dsZSB3aXRoIHN1Y2ggc2tpbGxzIG9mdGVuIGRvIHNvIGZvciBhIHZhcmlldHkgb2YgcmVhc29ucywgYW5kIGFuIGV4cGVydCBkaWFnbm9zaXMgY2FuIHNhdmUgeW91IGhvdXJzIG9mIGZydXN0cmF0aW9uLjwvcD48cD5UdXRvcnNUZWFjaC5jb20gd2FzIGRldmVsb3BlZCBhcyBhIGZyZWUgcmVzb3VyY2UgdG8gaGVscCBzdHVkZW50cyBjb25uZWN0IHdpdGggc3VwZXJpb3IgdHV0b3JzIGluIHRoZWlyIGFyZWFzLiAgTWFueSBzdWNoIHR1dG9ycyB3b3JrIHRocm91Z2ggdGVjaG5vbG9neSB0byBjcmVhdGUgb25saW5lIGxlc3NvbiBwbGFucywgZW1haWxzIGFuZCByZWFsLXRpbWUgY2hhdCB0dXRvcmlhbHMuICBXaGVuIHlvdSBmb3JlZ28gdGhvc2UgZXhwZW5zaXZlIGhvdXNlIHZpc2l0cywgeW91IGNhbiBsaXRlcmFsbHkgc2F2ZSBodW5kcmVkcyBvZiBkb2xsYXJzIG9uIHR1dG9yaW5nIGNvc3RzIG92ZXIgdGhlIGNvdXJzZSBvZiBhIHNjaG9vbCB5ZWFyITwvcD48cD5XaGF0ZXZlciB5b3VyIHdyaXRpbmcgbGV2ZWwsIHdlIGNhbiBoZWxwIGZpbmQgc29tZW9uZSB3aG9zZSBleHBlcnRpc2Ugc3VpdHMgeW91ciBuZWVkcy4gIFRoZSBzZWFyY2ggaXMgZnJlZSBmb3Igc3R1ZGVudHMgYW5kIG91ciB0dXRvcnMgcGF5IG9ubHkgYSBub21pbmFsIGZlZSB0byBnZXQgbGlzdGVkIGhlcmUsIHNvIHlvdSB3aWxsIG5ldmVyIGVuY291bnRlciBhbnkgdW5leHBlY3RlZCBmZWVzLiAgSWYgeW91IGFyZSB0aXJlZCBvZiB5b3VyIHBpZWNlcyBjb21pbmcgYmFjayBtYXJrZWQgdXAgd2l0aCByZWQgaW5rLCBhIHF1YWxpZmllZCB0dXRvciBtYXkgYmUgYSBncmVhdCBwbGFjZSB0byBzdGFydC48YnIgLz48L3A+DQo8cD48YSBocmVmPSJodHRwOi8vd3d3LmJiYm9ubGluZS5vcmcvY2tzLmFzcD9pZD0xMDUxMjE0MTExOTMzIiBvbmNsaWNrPSJ3aW5kb3cub3BlbignaHR0cDovL3d3dy5iYmJvbmxpbmUub3JnL2Nrcy5hc3A/aWQ9MTA1MTIxNDExMTkzMycsJ1JJQycsJ3dpZHRoPTYwMCxoZWlnaHQ9NDM4LHJlc2l6YWJsZT15ZXMsc2Nyb2xsYmFycz15ZXMsdG9vbGJhcj1ubyxtZW51YmFyPW5vLGxvY2F0aW9uPW5vLHN0YXR1cz1ubyxsZWZ0PTEwMCx0b3A9NDAsc2NyZWVuWD0xMDAsc2NyZWVuWT00MCcpO3JldHVybiBmYWxzZTsiPjxpbWcgc3JjPSJpbWFnZXMvQkJCU2VhbDMuZ2lmIiBhbHQ9IlRoZSBCZXR0ZXIgQnVzaW5lc3MgQnVyZWF1IFNlYWwiIHZzcGFjZT0iMTkiIGFsaWduPSJyaWdodCIgYm9yZGVyPSIwIi8+PC9hPjwvcD4NCjxwIGFsaWduPSJsZWZ0Ij48c3Ryb25nPlBhcmVudHMgYW5kIFN0dWRlbnRzOjwvc3Ryb25nPjwvcD4NCjxwIGFsaWduPSJsZWZ0Ij48YSBocmVmPSJsb2NhdGVUdXRvci5hc3B4Ij5DbGljayBoZXJlIHRvIEZpbmQgYSBUdXRvciBpbiB5b3VyIGFyZWEhPC9hPjxici8+RXZlcnkgcmVnaW9uLiBFdmVyeSBzdWJqZWN0LiBFdmVyeSBhY2FkZW1pYyBsZXZlbC4gQW5kIHRoZSBzZWFyY2ggZm9yIFR1dG9ycyBpcyBhbHdheXMgRlJFRSEgJm5ic3A7IDxhIGhyZWY9IkZsYXNoL3RvdXIuaHRtIiBvbmNsaWNrPSJ3aW5kb3cub3BlbignRmxhc2gvdG91ci5odG0nLCdSSUMnLCd3aWR0aD02MDAsaGVpZ2h0PTQzOCxyZXNpemFibGU9bm8sc2Nyb2xsYmFycz1ubyx0b29sYmFyPW5vLG1lbnViYXI9bm8sbG9jYXRpb249bm8sc3RhdHVzPW5vLGxlZnQ9MTAwLHRvcD00MCxzY3JlZW5YPTEwMCxzY3JlZW5ZPTQwJyk7cmV0dXJuIGZhbHNlOyI+Q2xpY2sgaGVyZSB0byBzZWUgYSBEZW1vPC9hPjxici8+PC9wPg0KPHAgYWxpZ249ImxlZnQiPjxzdHJvbmc+VHV0b3JzIGFuZCBFZHVjYXRvcnM6PC9zdHJvbmc+PC9wPg0KPHAgYWxpZ249ImxlZnQiPjxhIGhyZWY9Imh0dHBzOi8vd3d3LnR1dG9yc3RlYWNoLmNvbS9zaWduVXAuYXNweCI+Q2xpY2sgaGVyZSB0byBSZWdpc3RlciBhcyBhIFR1dG9yIT48L2E+PGJyLz5ObyBsb25nZXIgd2lsbCB5b3UsIHRoZSB0dXRvciwgbmVlZCB0byBmaW5kIG5ldyBzdHVkZW50cy4gTmV3IHN0dWRlbnRzIHdpbGwgZmluZCB5b3UuPGJyLz48YSBocmVmPSJmYXEuYXNweCI+PHNtYWxsPkxFQVJOIE1PUkUuLi48L3NtYWxsPjwvYT48L3A+DQpkAgcPZBYCAgEPDxYCHwAFBDIwMTBkZGR/VpP2kfOXZiQ6bnwjYk15PkTukw==";

$EVENTVALIDATION = "/wEWBALly9q5CQKMyrugBwKOhu66DgLz7ZDjD5DRTjhka66u5ECR4filq7BkSxxy";

curl_setopt ($ch, CURLOPT_POSTFIELDS, "left1_tbUser=myusername&left1_tbPassword=mypassword&__EVENTTARGET=".$EVENTTARGET."&__EVENTARGUMENT=".$EVENTARGUMENT."&__VIEWSTATE=".$VIEWSTATE."&__EVENTVALIDATION=".$EVENTVALIDATION);

 

// IMITATE CLASSIC BROWSER'S BEHAVIOUR : HANDLE COOKIES

curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');

 

# Setting CURLOPT_RETURNTRANSFER variable to 1 will force cURL

# not to print out the results of its query.

# Instead, it will return the results as a string return value

# from curl_exec() instead of the usual true/false.

curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

 

// EXECUTE 1st REQUEST (FORM LOGIN)

$store = curl_exec ($ch);

 

//print $store;

 

// SET FILE TO DOWNLOAD

curl_setopt($ch, CURLOPT_URL, 'https://www.tutorsteach.com/inquiry.aspx');

 

// EXECUTE 2nd REQUEST (FILE DOWNLOAD)

$content = curl_exec ($ch);

 

// CLOSE CURL

curl_close ($ch);

 

print $content;

 

?>

 

Now this should display the page that comes after we are logged into the site and visit this page: https://www.tutorsteach.com/inquiry.aspx

 

But it is showing this error:

"Object Moved

This object may be found here."

 

Can someone please help me? I am a novice in CURL.

 

thanks in advance.

 

FireFly

[roopurt18]

It sounds like the web server you're connecting to is returning a URL redirect.  You'll need to check the return code of the request (which I'm not sure how to do with cURL) and if it's one of the redirect codes (300, 301, others?) you'll need to hit whichever URL it is telling you it moved to.  Do this until you get a response code of 200 or something in the 4- or 5-hundreds.

[roopurt18]

Optionally cURL may have an option to perform redirects for you, so you might look into that.

[teamatomic]

Try:

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);

 

To get the header response use:

preg_match('*HTTP\/1\.1\s([0-9]{3})*is', $result, $status);

echo "$status[1]";

or for the entire header

print_r(curl_getinfo($ch));

 

 

HTH

Teamatomic

[fireflywins]

thanks a ton roopurt18 and HTH. I tried that line and now I am not getting that error and it is taking me to the redirect page.

 

But it is still showing me the page that comes without login which means that I am not logged in.

 

Can you please help?

 

This is the response of the server when used HTH's code to see header:

 

Array ( => https://www.tutorsteach.com/inquiry.aspx [content_type] => text/html; charset=utf-8 [http_code] => 200 [header_size] => 281 [request_size] => 122 [filetime] => -1 [ssl_verify_result] => 0 [redirect_count] => 0 [total_time] => 0.348745 [namelookup_time] => 3.0E-5 [connect_time] => 3.1E-5 [pretransfer_time] => 3.3E-5 [size_upload] => 0 [size_download] => 77633 [speed_download] => 222606 [speed_upload] => 0 [download_content_length] => 77633 [upload_content_length] => 0 [starttransfer_time] => 0.276132 [redirect_time] => 0 )

 

here I can see that http_code is 200 which means that it is right. Then when am I wrong.

 

Please help.

[roopurt18]

I can't help you much as I don't really know cURL.  Some of my PHP applications do communicate with web servers as if they were a browser though and I use the HttpRequest() object for that.  The documentation is a little buried so I'll link that to you:

http://php.he.net/manual/en/class.httprequest.php

 

More specifically the documentation for the send() method has some examples:

http://php.he.net/manual/en/function.httprequest-send.php

 

In my applications that use HttpRequest, I generally do the following:

1) Create a single HttpRequest object

2) Perform the initial POST to log in to a page

3) Check the response for any cookies from the web server and add them to the HttpRequest object

 

Now if I want to make future requests to the site, I can reuse the HttpRequest object that has the set cookies by doing the following:

1) Change the URL on the HttpRequest object

2) Change the method, GET or POST, as necessary

3) Add any POST data or files to upload

4) Call send

5) Check the response

 

I frequently use this method to write small export applications that upload data to web sites via HTTPS.

 

Following is a small code snippet from my code that you can adjust to set the object up:

<?php
class some_app{
    public function __construct() {
                // snippet...
	$http = new HttpRequest( $this->_url, HTTP_METH_POST );
	$http->addPostFields( array( 'client' => $this->_cfg['www']['client'],
		'key' => $this->_cfg['www']['key'] ) );
	$http->send();
	$cookies = $http->getResponseCookies();
	foreach( $cookies as $cookie ) {
		$http->addCookies( $cookie->cookies );
	}
	$json = json_decode( $http->getResponseBody() );
	$this->_continue = $json->Continue;
	$this->_http = $http;
                // snippet
    }

/**
 * Clears the HTTP request for another send operation.  Call this method from
         * elsewhere in your class to communicate with the web site you've logged into.
 *
 * @param string $url
 * @param array $post
 * @param array $postfiles
 * @param bool $send
 * @return stdClass|bool
 */
private function httpRequest( $url, $post, $postfiles = null, $send = false ) {
	$this->_http->setUrl( $url );
	$this->_http->setPostFields( $post );
	$this->_http->setPostFiles( $postfiles === null ? array() : $postfiles );
	if( $send !== true ) {
		return false;
	}
	$this->_http->send();
	return json_decode( $this->_http->getResponseBody() );
}
}
?>

 

Now I should mention some other things:

 

1) HttpRequest() is just a wrapper for cURL so you can accomplish all of the same with cURL.

 

2) In my case I control both the client (the program simulating the browser) and the server (the website).  Hence I know that I don't have any special handling such as URL redirects, accepting any cookies past the initial login, or worrying about session ID changes.

 

3) The php_http extension is required for HttpRequest() and can be difficult to locate.  I believe it is part of PECL, but I've found DLLs for Windows.

 

I hope that helps you some.