[SOLVED] Why is this not working?

[nutt318]

Does anyone know why this would not be working? It Did at one time but now it no longer works. Any suggestions would be awesome.

 

 

<?php
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "http://www.bloomberg.com/markets/commodities/energyprices.html");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);

$contents = curl_exec($ch);
curl_close($ch);

function find_values ($string, $page)
{
$string = preg_quote($string, '#');

// takes everything from the given string to end of row
preg_match("#$string(.*)</tr>#Us", $page, $match);

// Get the values from the row we found previously
preg_match_all("#<span[^>]*>([^<]*)</span>#s", $match[1], $values);

// Return the values	
return $values[1];
}

$find1 = find_values('Nymex Crude Future', $contents);
echo "Nymex Crude Future: Price = $find1[0], Change = $find1[1], & Change = $find1[2], Time = $find1[3]<br>";

$find2 = find_values('Dated Brent Spot', $contents);
echo "Nymex Heating Oil Future: Price = $find2[0], Change = $find2[1], & Change = $find2[2], Time = $find2[3]<br>";

$find3 = find_values('WTI Cushing Spot', $contents);
echo "Nymex RBOB Gasoline Future: Price = $find3[0], Change = $find3[1], & Change = $find3[2], Time = $find3[3]<br>")>


?>

[kenrbnsn]

What changed between the time it worked and when it stopped working?

 

Ken

[PFMaBiSmAd]

Also, define: "it no longer works"

 

That does not state any useful information for anyone in a forum to be able to help you.

 

What symptoms do you see? What do you see when you do a "view source" in your browser? What information is there in your web server log file?

 

Short answer: We only see the information you provide in your post. "it no longer works" won't get you a solution.

[nutt318]

At one time I would say 2 months ago it would pull the values of the Nymex Crude Future, basically there is a little table on the site that would update these fields througout the day. I just wanted to copy them and display them on my site. The only thing that may have changed was something on the remote website. So im guessing either their tables or something changed.

 

When I first used this code it would correctly get each value and store it so i could copy it on my site. For some reason now it no long gets anything at all.

 

When I view my source on my page it looks like

<td> </td>

 

I hope this helps, Thanks.

[nutt318]

*bump*

 

Any Ideas?

[revraz]

View their source, compare it to what you are looking for in your code.

[nutt318]

I did look at their source, it is as follows. I thought it looks ok but something is grabbing the info correctly or something.

 

Do you have any ideas revraz ?

 

 

 

<p>PETROLEUM ($/bbl)</p>
<div class="roundbottom">
<img style="display: none;" class="corner" src="http://images.bloomberg.com/r06/markets/bl.gif" height="4" width="4"></div>
</div>
</td>
</tr>
<tr>
<td align="center">
<table border="0" cellpadding="2" cellspacing="2" width="100%">
<tbody><tr align="right">

<td> </td><td><span class="tbl_txt">PRICE*</span></td><td><span class="tbl_txt">CHANGE</span></td><td><span class="tbl_txt">% CHANGE</span></td><td><span class="tbl_txt">TIME</span></td>
</tr>
<tr align="left" bgcolor="#ffffff">
<td><span class="tbl_txt">Nymex Crude Future</span></td><td align="right"><span class="tbl_num">92.44</span></td><td align="right"><span class="tbl_txt_green">1.81</span></td><td align="right"><span class="tbl_txt_green">2.00</span></td><td align="right"><span class="tbl_num">09:23</span></td>
</tr>
<tr>
<td align="left"><span class="tbl_txt">Dated Brent Spot</span></td><td align="right"><span class="tbl_num">92.80</span></td><td align="right"><span class="tbl_txt_green">1.34</span></td><td align="right"><span class="tbl_txt_green">1.47</span></td><td align="right"><span class="tbl_num">09:54</span></td>

</tr>
<tr bgcolor="#ffffff">
<td align="left"><span class="tbl_txt">WTI Cushing Spot</span></td><td align="right"><span class="tbl_num">92.46</span></td><td align="right"><span class="tbl_txt_green">1.83</span></td><td align="right"><span class="tbl_txt_green">2.02</span></td><td align="right"><span class="tbl_num">09:03</span></td>
</tr>
</tbody></table>
</td>
</tr>

[nutt318]

Any ideas anyone?

[nutt318]

Am I missing something, Ive looked at this code now for over 3 hours and still cannot find what is wrong. Or find a way to debug this problem. Can anyone else spot the problem?

[kenrbnsn]

I just did a quick test. When I use your curl lines as shown above, the value in the variable $contents is blank.

 

If I do

<?php
$contents = file_get_contents('http://www.bloomberg.com/markets/commodities/energyprices.html');
?>

 

I get the website HTML.

 

Try it that way.

 

Ken

[nutt318]

Ken, Thanks for the help. I tried what you said and now on my page it get this.

 

Warning: file_get_contents(): Circular redirect, aborting

Warning: file_get_contents(http://www.bloomberg.com/markets/commodities/energyprices.html): failed to open stream:

 

 

[nutt318]

Any ideas on why I get those errors on my page when I put that line of code in?

 

Also if anyone knows of the part of my code that seems to be incorrect can you give me an example on whats wrong?

 

Thanks.

[MadTechie]

They have added an "anti-leaching" protocal, you can get around it (one line of code infact) but i am guessing you don't have permission to use their data!

[nutt318]

ahh, where at is that in their source?

 

Thanks for the reply MadTechie

[MadTechie]

Not sure what you asked then..

 

if you had permission they would give you a link to use (probably an XML file) as your scrapping i assume you don't have permission.. so they have added a anti leaching procal you can get around it by adding one line of code to your existing code..

 

ahh whatever

 

try this

 

<?php
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "http://www.bloomberg.com/markets/commodities/energyprices.html");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);

curl_setopt($ch, CURLOPT_FILETIME, true);
curl_setopt($ch, CURLOPT_REFERER,""); 
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"); 

$contents = curl_exec($ch);
curl_close($ch);



$find1 = find_values('Nymex Crude Future', $contents);
echo "Nymex Crude Future: Price = $find1[0], Change = $find1[1], & Change = $find1[2], Time = $find1[3]<br>";

$find2 = find_values('Dated Brent Spot', $contents);
echo "Nymex Heating Oil Future: Price = $find2[0], Change = $find2[1], & Change = $find2[2], Time = $find2[3]<br>";

$find3 = find_values('WTI Cushing Spot', $contents);
echo "Nymex RBOB Gasoline Future: Price = $find3[0], Change = $find3[1], & Change = $find3[2], Time = $find3[3]<br>";


function find_values ($string, $page)
{
$string = preg_quote($string, '#');

// takes everything from the given string to end of row
preg_match("#$string(.*)</tr>#Us", $page, $match);

// Get the values from the row we found previously
preg_match_all("#<span[^>]*>([^<]*)</span>#s", $match[1], $values);

// Return the values	
return $values[1];
}

?>

 

 

EDIT: Opps wrong one! updated

[nutt318]

That worked, I added just this one line.

curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"); 

 

I see that you added two other lines, what do these lines do?

 

Thanks MadTechie

[MadTechie]

just clears the REFERER and examine the last requests filetime before requesting another uri

 

See human comments (hope it makes sense)

<?php
curl_setopt($ch, CURLOPT_FILETIME, true); //My Habbit
curl_setopt($ch, CURLOPT_REFERER,"");  //I belong here
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"); //Im not scrapping i am a browser
?>

[nutt318]

Awesome, thanks again for the help.

[MadTechie]

welcome review previous post for some detail