fopen - long URL issues

[iwpg]

Hello everyone, I just started having problems using the Bing Adcenter API with downloading reports in .zip format with fopen. There are no errors, so this is a bit frustrating. The file opens fine in a browser but not in the script. Please note that the file gets deleted after download, so it will produce a 404 error.

 

My code has been working for over a year, and I suspect that it's in the URL. I tested my code with a simple URL and did not have any issues. I am using PHP Version 5.3.27

 

Snippet of my code (the $reportDownloadUrl variable is automatically generated with the API, but I wanted to show the usage as well as syntax):

$reportDownloadUrl = "https://download.api.bingads.microsoft.com/ReportDownload/Download.aspx?q=3XL5uFR5BeF%2b1O9DOD6ZshdhvTn6vr9%2fUl9g9iXw3%2f%2f61Ly5IInt0O%2ba34CWMNxQBSV%2b4KolQhuTaalw3hZpvTcD3sq2uDDZw2c6a4Ud3%2f8V8OnlPwulQ29P3bbt9f2BpzQ5Ubcn%2bfH2oKfitYQsoj7R9o7dx%2fD6Bd2rmgwsRLbFLZiwz8NKh%2bcVjVEMKcPGFqcYDx%2fG9rxzeBTp00k3hcmnn2iGh2JnU2dyPCL%2fKxvKZzxZYKnjY7OztS2ONLZ68TCq1b%2fR7vWpr%2f8rkGxzSGEdBvK2adr5pFWw29HyrP0tmbE8jRRmgpbaH6SgNpJfmg%3d%3d";

	if (!$reader = fopen($reportDownloadUrl, 'rb')){ 
	throw new Exception("Failed to open URL " . $reportDownloadUrl . "."); 
	}

I really appreciate your help, it's been over 3 hours for me trying to figure out why this isn't working.

[mac_gyver]

There are no errors

 

 

do you have php's error_reporting set to E_ALL and display_errors set to ON so that any php detected errors will be reported and displayed?

 

what symptom are you getting that leads you to believe the code isn't working?

 

are you catching the exception from that code and what are you doing then in the catch code?

 

do you have php's output_buffing on, in your php.ini or in your script, and you are either redirecting or intentionally discarding the buffer so that any error output from the code is lost?

Edited December 15, 2013 by mac_gyver

[iwpg]

Thanks for the reply, I contacted SoftLayer and they were pretty stumped as well. The firewall and Mod_Security is off, and the new server is pretty much a clone to my old one. That's all that they could come up with. To answer your questions:

 

do you have php's error_reporting set to E_ALL and display_errors set to ON so that any php detected errors will be reported and displayed?

YES

 

what symptom are you getting that leads you to believe the code isn't working?

Printing contents returns empty, no file gets written. I have confirmed that this works by testing other .zip files on my other URLs by downloading them with this same script.

 

are you catching the exception from that code and what are you doing then in the catch code?

The response is successful

 

do you have php's output_buffing on, in your php.ini or in your script, and you are either redirecting or intentionally discarding the buffer so that any error output from the code is lost?

Yes, output_buffering is on, and I was flushing the buffer. I disabled this to test and had the same result.

 

My only suspicion at this point is that my new IP is blocked by Microsoft's server. I sent a request to them to check on it.

 

Thank you!

 

Mike

[iwpg]

Ok, after a lot of frustration, I believe it has something to do with the SSL version. Enabling curl_setopt($ch, CURLOPT_SSLVERSION, 3); now returns a response code of 200, instead of 0. It does not appear that mod_ssl is installed on the current version of Apache.

[iwpg]

Still not the solution (Argghh). I reverted back to Apache 2.2 due to mod_ssl not compiling correctly with 2.4.7 (a known bug issue). So far, my best guess is that I'm being blocked by their server, even though my API account is active and in good standing. Any ideas are truly welcome. Please see my posts on Microsoft's API support board for an update.

[Barand]

Might be worth checking if you can open files using https

<?php
$w = stream_get_wrappers();
echo '<pre>';
echo 'openssl: ',  extension_loaded  ('openssl') ? 'yes':'no', "\n";
echo 'http wrapper: ', in_array('http', $w) ? 'yes':'no', "\n";
echo 'https wrapper: ', in_array('https', $w) ? 'yes':'no', "\n";
echo 'wrappers: ', print_r($w,1);
echo '</pre>';
?>

[iwpg]

Hi Barand,

 

I am getting the following result. The only difference between my old server is that ldap and ldaps are active on the one that's not working.

openssl: yes
http wrapper: yes
https wrapper: yes
wrappers: Array
(
    [0] => compress.zlib
    [1] => dict
    [2] => ftp
    [3] => ftps
    [4] => gopher
    [5] => http
    [6] => https
    [7] => imap
    [8] => imaps
    [9] => ldap
    [10] => ldaps
    [11] => pop3
    [12] => pop3s
    [13] => rtsp
    [14] => smtp
    [15] => smtps
    [16] => telnet
    [17] => tftp
    [18] => php
    [19] => file
    [20] => glob
    [21] => data
    [22] => phar
    [23] => zip
)

[jazzman1]

As far as I know if cURL return 0 that's just fine

 

Did you change the server platform or only php version?

 

What server are you using right now?   

[iwpg]

Hi Jazzman1,

 

I moved servers, but I pretty much have the same setup. I downgraded Apache and PHP to mirror the old one, but still no luck. This is the code that I used to test, it should return the 404 page that is there (the actual zip file gets deleted after 5 minutes). This works on my old server fine. Changing the URL and removing the "download" subdomain works, it returns the 500 error page.

Attempted to download content from https://download.api.bingads.microsoft.com/ReportDownload/Download.aspx and wrote it to <a href="log_microsoft.txt">this text file</a>

<?php
$fileName = "log_microsoft.txt";
$downloadURL = "https://download.api.bingads.microsoft.com/ReportDownload/Download.aspx";
$handleOut = fopen($fileName, "w");
$handleIn = fopen($downloadURL,"r");

	while (!feof($handleIn)){
	$content = fread($handleIn, 8192);
	print_r ($content);
	fwrite($handleOut,$content);
	}

fclose($handleIn);
fclose($handleOut);
?>

[jazzman1]

Now, try what mac_gyver suggested you at the first reply. 

 

Open up the php.ini file and find and change the line output_buffer = 4096 to output_buffer = Off and restart the web server. 

 

Or try this:

<?php
ini_set('display_errors',1);
ini_set('display_startup_errors',1);
ini_set('output_buffering','Off');
error_reporting(-1);
$fileName = "log_microsoft.txt";
$downloadURL = "https://download.api.bingads.microsoft.com/ReportDownload/Download.aspx";
$handleOut = fopen($fileName, "w");
$handleIn = fopen($downloadURL,"r");

    while (!feof($handleIn)){
    $content = fread($handleIn, 8192);
    print_r ($content);
    fwrite($handleOut,$content);
    }

fclose($handleIn);
fclose($handleOut);

You still doesn't give me an answer - what server are you using (operating system)?

[iwpg]

No error, nothing. Just an empty screen. I turned off output buffering, and also ran the code that you gave me.

I am using Centos 6.4 (64 bit) with Apache 2.2.26, PHP 5.4.23

[jazzman1]

 

I am using Centos 6.4 (64 bit) with Apache 2.2.26, PHP 5.4.23

 

Me too. I'm a big linux fan

 

Does SeLinux have enforcing  status at the moment?

 

Can you give me the output of:

sestatus

And this one too:

getsebool -a | grep httpd

Edited December 18, 2013 by jazzman1

[iwpg]

Hi Jazzman, the state is "disabled".

[jazzman1]

Hm....no idea...sorry for that.

 

EDIT:: Are you able to read the logs of the CentOS machine?

Edited December 18, 2013 by jazzman1

[iwpg]

Me neither, and I really appreciate Barand's, Mac_Gyver's and your help. I cannot confirm if the server is blocked or not. This is Microsoft we're talking about, so I decided to order a temporary machine to test this, instead of possibly waiting weeks or months for no answer. It would be nice to know if this is server/ip/datacenter related or not. $115 bucks for a month's of extra hosting is worth it at this point to find out where to look next.

[kicken]

Have you tried downloading these URL's without apache or PHP in the picture? Use the cURL command line client or wget to download them from the command line. If those tools also fail, then you maybe a looking at some kind of IP block or routing issue. If they work however, then you have an issue either in your code or in your PHP/Apache setup.

[iwpg]

Hi Kicken, good idea. Curl and wget both resolve! This helps out quite a bit. Thanks!

[iwpg]

Hallelujah!!! - The problem is solved! It certainly looks like OpenSSL was the issue, something to do with permissions/RPM on the Centos server.

http://httpd.apache.org/docs/current/platform/rpm.html Last night Cpanel made an update to the PHP Pear package (cpanel-php53-PEAR-1.9.4-3.cp1136.src.rpm) and it fixed the issues related to cURL, but not fopen via SSL. At least I'm able to download via cURL now.

 

On the new server that I set up, I was able to generate these errors (for whatever reason, the main server did not generate any output, with error reporting enabled).

Warning:SSL: Connection reset by peer

Warning: fopen(): Failed to enable crypto

 

It sounds like 64-bit versions of Centos/PHP is known for issues with OpenSSL based on some articles that I came across.

http://stackoverflow.com/questions/18098258/64-bit-linux-ubuntu-and-openssl-issue-could-not-read-symbols-bad-value

 

I hope that this can help prevent somebody for not going through what I did! Thanks for all the help with this, almost in tears now! LOL