Tumblr Curl Not Working?

Lukeidiot · November 4, 2012

Is it possible for a site to block cURL?

Here's my code that was working about a month ago:

<?php

$tumblr_email = 'user@user.com';
$tumblr_password = 'pass';

$request_data = http_build_query(
   array(
       'email'     => $tumblr_email,
       'password'  => $tumblr_password
   )
);


$c1 = curl_init('http://www.tumblr.com/login');
curl_setopt($c1, CURLOPT_POST, true);
curl_setopt($c1, CURLOPT_POSTFIELDS, $request_data);
curl_setopt($c1, CURLOPT_RETURNTRANSFER, true);
curl_setopt($c1, CURL_COOKIEFILE, 'cookie.txt');  
curl_setopt($c1, CURLOPT_COOKIEJAR, 'cookie.txt');  
curl_setopt($c1, CURLOPT_FOLLOWLOCATION,true);

$result1 = curl_exec($c1);

echo $result1;

curl_close($c1);

?>

MMDE · November 4, 2012

It's not "possible" to block cURL, by that I mean if you can contact it with your normal web browser, you can do it with cURL too.

You must pretend to be a netbrowser, that's the solution.

You can see here how explicit the way they verify you are "real" can be:

http://forums.phpfre...equired-inside/

I also explain how I figured out the problem in that case, if you can't figure it out then give me a test account and I will help you with it.

Most likely you just need to send user-agent data, but can also be they do something else to verify the login.

Edited November 4, 2012 by MMDE

Lukeidiot · November 4, 2012

It's not "possible" to block cURL, by that I mean if you can contact it with your normal web browser, you can do it with cURL too.

You must pretend to be a netbrowser, that's the solution.

You can see here how explicit the way they verify you are "real" can be:

http://forums.phpfre...equired-inside/

I also explain how I figured out the problem in that case, if you can't figure it out then give me a test account and I will help you with it.

Most likely you just need to send user-agent data, but can also be they do something else to verify the login.

I have PMed you a username/password for a test account.

However, when testing the cURL, I find it says "An error has occured." instead of "Invalid username or password.".

I am still a little lost when trying to find the missing key to login here.

I also tried adding a user-agent, which did not help as well.

Edited November 4, 2012 by Lukeidiot

MMDE · November 5, 2012

It's not an easy site to do this with, that's for sure...

Are you sure you can't just use their API?

function tumblr($url, $email, $password){
$post = http_build_query(
 array(
  'email' => $email,
  'password'  => $password
 )
);
$ch = curl_init('http://www.tumblr.com/login');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 1);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
//echo $data;
curl_close($ch);
preg_match_all('|Set-Cookie: (.*);|U', $data, $matches);   
$cookie = implode('; ', $matches[1]);
echo $cookie;
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 1);
curl_setopt($ch, CURLOPT_HEADER, true);
return curl_exec($ch);
}

That is what I currently have. The first part goes just fine, but the second doesn't seem to work.

I'm posting this so someone else can give it a shot too.

MMDE · November 5, 2012

I got it to work now!!

Will post the code in a second!

Just need to make it so you can actually use it!

It's a bit strange system they got, but I'm more and more narrowing it down, and trying to make it all dynamic and not just working for me.

I believe you need to send 4 postfields the second time.

Edited November 5, 2012 by MMDE

Lukeidiot · November 5, 2012

I got it to work now!!

Will post the code in a second!

Just need to make it so you can actually use it!

It's a bit strange system they got, but I'm more and more narrowing it down, and trying to make it all dynamic and not just working for me.

I believe you need to send 4 postfields the second time.

Very nice.

MMDE · November 5, 2012

Very nice.

Nope, because that cookie I fetched there is useless. I'm not even sure why they use it anymore. I think it might have something to do with their old system.

Wasted a lot of time on that cookie. lol

It seems like they use google's recaptcha. It seems to be working in the background, giving you the values when you load the page or something. So it might be possible to write a work-around that.

Edited November 5, 2012 by MMDE

Lukeidiot · November 5, 2012

Nope, because that cookie I fetched there is useless. I'm not even sure why they use it anymore. I think it might have something to do with their old system.

Wasted a lot of time on that cookie. lol

It seems like they use google's recaptcha. It seems to be working in the background, giving you the values when you load the page or something. So it might be possible to write a work-around that.

Ah. Do you think its still possible to make work?

MMDE · November 5, 2012

Ah. Do you think its still possible to make work?

Yes, I'm pretty sure I can, because I just got the values I wanted from the site.

Will probably be done in some few minutes!

MMDE · November 5, 2012

<?php
function tumblr($url, $email, $password){
$pa = array(
'user[email]' => $email,
'user[password]' => $password,
'user[age]' => 50,
'user[tos]' => 1
);
$post = http_build_query($pa);
$ch = curl_init('http://www.tumblr.com/login');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
curl_close($ch);
$site = new DOMDocument();
@$site->loadHTML($data);
$inputs = $site->getElementsByTagName('input');
foreach($inputs AS $input){
if($input->hasAttribute('name')){
switch($input->getAttributeNode('name')->value){
case 'recaptcha_public_key':
$pa['recaptcha_public_key'] = $input->getAttributeNode('value')->value;
break;
case 'recaptcha_response_field':
$pa['recaptcha_response_field'] = $input->getAttributeNode('placeholder')->value;
break;
case 'http_referer':
$pa['http_referer'] = $input->getAttributeNode('value')->value;
break;
case 'form_key':
$pa['form_key'] = $input->getAttributeNode('value')->value;
break;
}
}
}
$post = http_build_query($pa);
$ch = curl_init('http://www.tumblr.com/login');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
curl_close($ch);
preg_match_all('|Set-Cookie: (.*);|U', $data, $matches);
$cookie = implode('; ', $matches[1]);
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
return curl_exec($ch);
}
$url = 'http://www.tumblr.com/dashboard';
$email = 'the email';
$password = 'the password';
$data = tumblr($url, $email, $password);
echo $data;
?>

I was able to load the dashboard using this code and the account information you gave me!

I know I could have broken it down into a far smaller code, but this works.

Edited November 5, 2012 by MMDE

MMDE · November 5, 2012

<?php


function get_data($url, $post=null, $header=false, $cookie=null){
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
if($header) curl_setopt($ch, CURLOPT_HEADER, true);
if($cookie) curl_setopt($ch, CURLOPT_COOKIE, $cookie);
return curl_exec($ch);
}


function add_postfields($data, &$post){
$site = new DOMDocument();
@$site->loadHTML($data);
$inputs = $site->getElementsByTagName('input');
foreach($inputs AS $input){
 if($input->hasAttribute('name')){
  switch($input->getAttributeNode('name')->value){
   case 'recaptcha_public_key':
 $post['recaptcha_public_key'] = $input->getAttributeNode('value')->value;
 break;
   case 'recaptcha_response_field':
 $post['recaptcha_response_field'] = $input->getAttributeNode('placeholder')->value;
 break;
   case 'http_referer':
 $post['http_referer'] = $input->getAttributeNode('value')->value;
 break;
   case 'form_key':
 $post['form_key'] = $input->getAttributeNode('value')->value;
 break;
  }
 }
}
}


function get_cookies($data){
preg_match_all('|Set-Cookie: (.*);|U', $data, $matches);   
return implode('; ', $matches[1]);
}


function tumblr($url, $email, $password){
$post = array(
 'user[email]' => $email,
 'user[password]'  => $password,
 'user[age]' => 50,
 'user[tos]' => 1
);
add_postfields(get_data('http://www.tumblr.com/login', http_build_query($post)), $post);
$cookie = get_cookies(get_data('http://www.tumblr.com/login', http_build_query($post), true), $post);
return get_data($url, null, false, $cookie);
}
$url = 'http://www.tumblr.com/dashboard';
$email = 'the email';
$password = 'the password';
$data = tumblr($url, $email, $password);
echo $data;
?>

This one should look better.

Edited November 5, 2012 by MMDE

Lukeidiot · November 6, 2012

Thanks MMDE you are very talented with cURL.

player001 · June 11, 2013

Hi,

Sorry to dig that thread up, but it seems that the really nice piece of code made by MMDE does not work anymore.

Tumblr has added a new postfield : tumblelog[name], between password and age. But even when modifying the 'tumblr' function accordingly, the login is never successful. I've also tried to use 'https' instead of 'http', but it does not change a thing.

Could anyone have a look at the changes that may have made this code obsolete ? They must not be that big... Maybe it's the cookie. If you do not want to bother checking the whole source code, could you explain where I have to look at to change it ? I don't know much about cookies...

Thank you !

PS : i can PM a test account if needed

Edited June 11, 2013 by player001

player001 · June 11, 2013

Some other modifications in function add_postfields, though i can't make the code work yet :

case 'recaptcha_public_key':
$post['recaptcha_public_key'] = $input->getAttributeNode('value')->value;
break;
case 'recaptcha_response_field':
$post['recaptcha_response_field'] = $input->getAttributeNode('placeholder')->value;
break;
case 'form_key':
$post['form_key'] = $input->getAttributeNode('value')->value;
break;
case 'seen_suggestion':
$post['seen_suggestion'] = $input->getAttributeNode('value')->value;
break;
case 'used_suggestion':
$post['used_suggestion'] = $input->getAttributeNode('value')->value;
break;

('http_referer' dispareared and two other fields - used/seen suggestion - were added)

Edited June 11, 2013 by player001

MarkMasic · August 11, 2014

I know this is an old thread, but after much searching and frustration myself to find any info that is recent, I thought I'd post my variation of the above code which as of the time of this post works.

There are four variables at the bottom in CAPS which you need to change.

<?php

function get_data($url, $post=null, $header=false, $cookie=null){

$ch = curl_init($url);

curl_setopt($ch, CURLOPT_POST, true);

curl_setopt($ch, CURLOPT_POSTFIELDS, $post);

curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER,false);

curl_setopt($ch, CURLOPT_SSL_VERIFYHOST,false);

curl_setopt($ch, CURLOPT_CONNECTTIMEOUT,30);

curl_setopt($ch, CURLOPT_TIMEOUT,60);

curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/12.0.702.0 Safari/534.24');