problem in reading an external web page

[AladdinMhaimeed]

Hi,

I'm trying to read the page at this address:

http://www.tadawul.com.sa/wps/portal/!ut/p/.cmd/cs/.ce/7_0_A/.s/7_0_4AI/_s.7_0_A/7_0_4AI

to my php page,

unfortunately, I tried every known way and failed with all,

I tried fopen, stream files, curl, sockets.

If it has benefit I can show the code that I used for every trail,

[MadTechie]

$data = file_get_contents("http://www.tadawul.com.sa/wps/portal/!ut/p/.cmd/cs/.ce/7_0_A/.s/7_0_4AI/_s.7_0_A/7_0_4AI");

or

cURL

 

[AladdinMhaimeed]

thank you for your rapid reply,

unfortunatelly, I tried it and failed,

it returns a CGI Error,

Have you tired it yourself?

thank you,

[jaymc]

Show the code...

 

file_get_contents doesnt work over http I believe

 

Cant see why a socket or curl isnt working providing you are setting a browser agent etc

[AladdinMhaimeed]

here is the code for curl:

 

<?php
$url = "http://www.tadawul.com.sa/wps/portal/!ut/p/.cmd/cs/.ce/7_0_A/.s/7_0_4AI/_s.7_0_A/7_0_4AI";
$ch = curl_init($url);    // initialize curl handle
//curl_setopt($ch, CURLOPT_URL, $url); // set url to post to
//curl_setopt($ch, CURLOPT_FAILONERROR, 1);              // Fail on errors
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);    // allow redirects
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1); // return into a variable
curl_setopt($ch, CURLOPT_PORT, 80);            //Set the port number
curl_setopt($ch, CURLOPT_TIMEOUT, 15); // times out after 15s
curl_setopt($ch, CURLOPT_HEADER, true);

$document = curl_exec($ch);

echo "<pre>".$document."</pre>";
exit;
?>

 

and here is the one for socket:

<?php
$fp = fsockopen("http://www.tadawul.com.sa/wps/portal/!ut/p/.cmd/cs/.ce/7_0_A/.s/7_0_4AI/_s.7_0_A/7_0_4AI", 80, $errno, $errstr, 300);
if (!$fp) {
    echo "$errstr ($errno)<br />\n";
} else {
    $out = "GET / HTTP/1.1\r\n";
    $out .= "Host: http://www.tadawul.com.sa/wps/portal/!ut/p/.cmd/cs/.ce/7_0_A/.s/7_0_4AI/_s.7_0_A/7_0_4AI\r\n";
    $out .= "Connection: Close\r\n\r\n";

    fwrite($fp, $out);
    while (!feof($fp)) {
        echo "<pre>". fgets($fp, 128)."</pre>";
    }
    fclose($fp);
}
?> 

[MadTechie]

its probably blocking access from scrapping access, do you have permission to scrap ?

if you they need to allow your server IP

[AladdinMhaimeed]

thank you very much,

actually I dont have a permission,

but when I try to scrap this page on the same domain and server, i have no problem, and read it in almost whatever method.

http://www.tadawul.com.sa/Resources/Reports/DailyList_ar.html

 

how can I figure out whether I need a permission or not?

if the answer is that I need one, then can I use a work around? after all, the page is open to read via http, and there must be a way to read it.

what do you think,

 

[MadTechie]

Tadawul claims all copyright and intellectual property rights in all content on this website and in the underlying coding, or has the permission of the owner of those rights to use the content on this website.

 

The names of all products and services mentioned on this website are the trademarks of their owners.

 

your need to speak to them..

 

can it be done, without their permission,

yes.. but i won't be party to that.. (sorry, if i blocked it mean i don't want leachers)

 

proof of concept

 

[AladdinMhaimeed]

I respect that, but can you at least tell me what have they done to prevent regular reading methods?

thank you,

[MadTechie]

its basically a .htaccess mod,

it checks a few things,

 

Oh @@@@ it..

 

they can still block your server IP, so don't leach like mad..

 

<?php
$URL = "http://www.tadawul.com.sa/wps/portal/!ut/p/.cmd/cs/.ce/7_0_A/.s/7_0_4AI/_s.7_0_A/7_0_4AI";
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 1); // Get the header
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // Allow redirection
curl_setopt($ch, CURLOPT_COOKIEJAR, "/tmp/cookie");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FILETIME, true);

curl_setopt($ch, CURLOPT_URL, $URL);
curl_setopt($ch, CURLOPT_REFERER,""); //Clear REFERER
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"); //Fake Client

$data = curl_exec($ch);
curl_close($ch); 

echo $data;
?>

 

[AladdinMhaimeed]

well, thank you very much indeed,

I will use it only to show my customer that it could be dont technically,

and then the permission and the copyright issues will by his own business,

any way, if you don't accept me to use it for this purpuse, I guaranty to you I will not use it.

[MadTechie]

use it as needed but remember, if you was the admin of that site and wanted block leachers, its a pain when someone finds a way to continue, theirs a few more things they can do to protect themselfs but still..

they have something inplace to make it hard to leach which to me means they don't want leachers..

 

ok i yapping now..

use as needed

[AladdinMhaimeed]

thank you, i respect all that