Jump to content


Photo

CURL question follow url and pull next url from url bar


  • Please log in to reply
3 replies to this topic

#1 Trevors

Trevors
  • Members
  • PipPip
  • Member
  • 11 posts

Posted 05 August 2006 - 07:20 PM

Hi all,
Im curious if CURL can follow a link like this "http://www.blogcatal...og-idiota.html" then when it redirects it pulls the real url and saves it.
I was gonna crawl whole http://www.blogcatalog.com/ but they got redirect urls which i dont want.
Would really appreciate abit of help

Best Regards
Trevor

#2 shoz

shoz
  • Staff Alumni
  • Advanced Member
  • 600 posts

Posted 06 August 2006 - 01:14 AM

http://www.phpfreaks....html#msg387319

#3 Trevors

Trevors
  • Members
  • PipPip
  • Member
  • 11 posts

Posted 06 August 2006 - 09:34 AM

Thanks mate :D really appreciate it

#4 shoz

shoz
  • Staff Alumni
  • Advanced Member
  • 600 posts

Posted 06 August 2006 - 11:46 AM

I think I slightly misinterpreted your post even though the code snippet I linked you to does also have the answer. I thought you just wanted to get the address of the final url, but I realize now that you want the content (I think).

You may have figured it out but know that the only curl option you need to set is CURLOPT_FOLLOWLOCATION for curl to follow the redirects.
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
<?php
$ch = curl_init();
$url = 'http://example.com/website/hello/329472';

curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

$r = curl_exec($ch);
$url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch)

//$r has the content
//$url has the address
Note the removal of the NOBODY option.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users