Memory limit issues when downloading a large file to host.

shlomikalfa · December 21, 2008

Hi

i am trying to download a large file to my host [100+MB]

but it is always stuck when it reaches my hosts memory limit [40MB]

my host doesn't permit changing the memory limit...

any way around it maybe ?!

I just can't seem to understand why can't i just write the data into a file and go on.... why do i have to stack the entire data into the memory ?!?

shlomikalfa · December 21, 2008

should i maybe use:

while (!feof($Socket)){
$fs = @fopen($saveToFile, "a");
.....
@flock($fs, LOCK_EX);
$bytesSaved = fwrite($fs, $dopdata);
....
@flock($fs, LOCK_UN);
@fclose($fs);
}
[code]

To open/Write/Close inside my loop?!?
How do i clean the memory after writing the data to file ?!?

redarrow · December 21, 2008

not sure here but you might like to use the following code.

don't forget just because u haven't got access to the php.ini you

still got some control over php using ini_set() function.

url ini_set()

http://it.php.net/ini_set

$x = ini_set('memory_limit', '64M');

shlomikalfa · December 21, 2008

i just need to know how to i get a 'fsockopen' then i 'fgets' and write it a file... after i've written it to a file i need to FREE the memory holding that data i've just wrote to the file.... that's alll

i have already put this:

@ini_set("memory_limit", "1024M");
@ini_set("output_buffering","off");

it has no effect.... also tried putting my own php.ini / htaccess.... it doesn't work.... the host doesn't allow changing the memory_limit!

>> i need to know how can i download a huge file [over 100MB] without having to keep the entire size of it in the memory!

redarrow · December 21, 2008

For download the big files (more than 8MB), you must used ob_flush()

because the function flush empty the Apache memory and not PHP memory.

And the max size of PHP memory is 8MB, but ob_flush is able to empty the

PHP memory.

header('Content-Type: application/force-download');

header ("Content-Length: " . filesize($file));

header ("Content-Disposition: attachment; filename=$theFileName");

$fd = fopen($file, "r");

while(!feof($fd))

{

echo fread($fd, 4096);

ob_flush();

}

shlomikalfa · December 21, 2008

i will try that thanks...

meanwhile here is my code snippet:

function geturl($host, $port, $url, $referer = 0, $cookie = 0, $post = 0, $saveToFile = 0, $proxy = 0, $pauth = 0, $scheme = "http") {
global $nn, $lastError, $PHP_SELF, $AUTH, $HTTPcode, $progressbar_via_time, $agent, $sleep_time, $sleep_count, $max_speed, $title_update, $ResumeMode;
global $is_header;

if (($post !== 0) && (($scheme == "http") || ($scheme == "https")))
{
	$method = "POST";
	$postdata = formpostdata($post);
	$length = strlen($postdata);
	$content_tl = "Content-Type: application/x-www-form-urlencoded".$nn."Content-Length: ".$length.$nn;
}
	else
{
	$method = "GET";
	$postdata = "";
	$content_tl = "";
}

if ($cookie)
{
	if (is_array($cookie))
		{
			for( $h=0; $h<count($cookie); $h++)
				{
					$cookies.="Cookie: ".trim($cookie[$h]).$nn;
				}
		}
			else
		{
			$cookies = "Cookie: ".trim($cookie).$nn;
		}
}

$referer = $referer ? "Referer: ".$referer.$nn : "";

if($proxy)
{
	list($proxyHost, $proxyPort) = explode(":", $proxy);
	$url = $scheme."://".$host.":".$port.$url;
}

$auth = ($AUTH["use"] === TRUE) ? "Authorization: Basic ".$AUTH["str"].$nn : "";
$proxyauth = ($pauth) ? "Proxy-Authorization: Basic $pauth".$nn : "";

$Range = "";
if($saveToFile){
if ($ResumeMode and file_exists($saveToFile)){
	$FileSize = filesize($saveToFile);
	$Range = "Range: bytes=".$FileSize."-".$nn;
	clearstatcache();
}
}

$zapros=
$method." ".str_replace(" ", "%20", $url)." HTTP/1.1".$nn.
"Host: ".$host.$nn.
"User-Agent: $agent".$nn.
"Accept: */*".$nn.
"Accept-Language: en-us;q=0.7,en;q=0.3".$nn.
"Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7".$nn.
"Connection: Close".$nn.
$Range.
$auth.
$proxyauth.
$referer.
$cookies.
$content_tl.$nn.$postdata;

//echo "<pre>$zapros</pre>\n";

$fp = @fsockopen($proxyHost ? $proxyHost : ((($scheme == "https") && !$proxy) ? "ssl://" : "").$host, $proxyPort ? $proxyPort : $port, &$errno, &$errstr, 15);

if (!$fp)
{
	html_error("Error open socket [".($proxyHost ? $proxyHost : $host).':'.($proxyPort ? $proxyPort : $port)."]<br>$errstr");
}

stream_set_timeout($fp, 300);

if($errno || $errstr)
{
	$lastError = $errstr;

	return false;
}

fputs($fp,$zapros);
fflush($fp);
$timeStart = getmicrotime();

$size_buff=256;

while(!feof($fp))
{
	 	$data=fgets($fp,$size_buff);
	fflush($fp);
		if ($data === false) {break;}

 	if (($sleep_count !== false) && ($sleep_time !== false) && is_numeric($sleep_time) && is_numeric($sleep_count) && ($sleep_count > 0) && ($sleep_time > 0))
 		{
 			$local_sleep--;
 			if ($local_sleep == 0)
 				{
 					usleep($sleep_time);
 					$local_sleep=$sleep_count;
 				}
		}

	if($saveToFile)
		{
			if($headersReceived)
				{
					$fs = @fopen($saveToFile, "a");
					@flock($fs, LOCK_EX);
					$bytesSaved = fwrite($fs, $data);
					if($bytesSaved > -1)
						{
							$bytesReceived += $bytesSaved;

							if ($bytesTotal && ($bytesReceived > $bytesTotal))
								{
									$bytesReceived=$bytesTotal;
									fseek($fs,$bytesTotal);
									ftruncate($fs,$bytesTotal);

									fclose($fp);
									unset($fp);

									break;
								}
						}
							else
						{
							echo "It is not possible to carry out a record in File".basename($saveToFile);
							return false;
						}
						@flock($fs, LOCK_UN);
						@fclose($fs);
    
					$time = getmicrotime() - $timeStart;
					$chunkTime = $time - $lastChunkTime;
					$chunkTime = $chunkTime ? $chunkTime : 1;
					$lastChunkTime = $time;
					$speed = round(($bytesReceived-$last) / 1024 / $chunkTime, 2);
					$last = $bytesReceived;

					if ((is_numeric($progressbar_via_time)  && (time() - $lastsent > $progressbar_via_time)) || (!is_numeric($progressbar_via_time) && ($bytesReceived > $lastReceive + $chunkSize)))
						{
							$time2 = getmicrotime() - $timeStart;
							$chunkTime2 = $time2 - $lastChunkTime2;
							$chunkTime2 = $chunkTime2 ? $chunkTime2 : 1;
							$mspeed=round(($bytesReceived-$lastReceive) / 1024 / $chunkTime2, 2);
							$mspeed = $mspeed ? $mspeed : 1;
							$lastChunkTime2 = $time2;

							$lastsent=time();
							$lastReceive=$bytesReceived;

							$received = bytesToKbOrMb($bytesReceived);

	              			if (!$bytesTotal || ($bytesReceived >= $bytesTotal))
	              				{
									$percent = "100%";
									$out='';
								}
									else
								{
									$percent =round($bytesReceived / $bytesTotal * 100, 2);

									$out=($bytesTotal-$bytesReceived) / ($mspeed * 1024);
									$out=sec2time(round($out,2));
								}

							echo "<script>pr('".$percent."', '".$received."', '".$mspeed."','".$out."')</script>\r\n";
						}

				}
					else
				{
					$tmp .= $data;
					if (strstr($tmp, "\n\n")) {$det = "\n\n";}
					elseif (strstr($tmp, $nn.$nn)) {$det = $nn.$nn;}

					if($det)
						{
							$size_buff=4096;

							list($header,$dopdata) = explode($det, $tmp);
							$headersReceived = true;

							$headers=explode( (($det == "\n\n") ? "\n" : $nn), $header);
							list($prototype,$HTTPcode,$httpres)=explode(' ',$headers[0]);

							switch ($HTTPcode)
								{
									case 200:
									case 201:
									case 202:
									case 203:
									case 204:
									case 205:
									case 206:
									break; //Resuming!

									case 300:
									case 301:
									case 302:
									case 303:
									case 304:
									case 305:
									case 307:
										$redirect = cut_str($header, "Location:", "\n");

										if ($redirect)
											{
												$lastError = "$HTTPcode Info!! it is redirected to [".$redirect."] On the course of events, the reference became obsolete. So start from the beginning...<br><br><a href=\"".$PHP_SELF."\" style=\"color: #0000FF;\">Main</a>";
											}
												else
											{
												$lastError = "$HTTPcode Error!! it is redirected. On the course of events, the reference became obsolete. So start from the beginning...<br><br><a href=\"".$PHP_SELF."\" style=\"color: #0000FF;\">Main</a>";
											}
										return $header;
									break;

									case 401:
										$lastError = "401 Error!! This site requires authorization. For the indication of login and password of access it is necessary to use similar url:<br>http://<b>login:password@</b>www.site.com/file.exe";
										return FALSE;
									break;

									case 403:
										$lastError = "403 Error!! Access denied";
										return FALSE;
									break;

									case 407:
										$lastError = "407 Error!! Proxy Authentication Required';";
										return FALSE;
									break;

									case 404:
										$lastError = "404 Error!! Necessary file not discovered on server";
										return FALSE;
									break;

									case 400:
									case 401:
									case 402:
									case 405:
									case 406:
									case 407:
									case 408:
									case 409:
									case 410:
									case 411:
									case 412:
									case 413:
									case 414:
									case 415:
									case 416:
									case 417:
									case 422:
									case 423:
									case 424:
									case 426:
									case 500:
									case 501:
									case 502:
									case 503:
									case 504:
									case 505:
									case 506:
									case 507:
									case 510:
										$lastError="Error!! ".HTTPStatus($HTTPcode);
										return FALSE;
									break;

									default:
										$lastError="Error!! Unknown HTTP Status: ".pre($header);
										return FALSE;
									break;
								}

							$bytesTotal = trim(cut_str($header, "Content-Length: ", "\n"));
							$bytesTotal = $bytesTotal ? $bytesTotal : 0;

							// Make sure we don't over-write the file.
							if ($FileSize == $bytesTotal and $bytesTotal > 0 and $FileSize > 0){
								$lastError="File download is done!".pre($header);
								return FALSE;}


							$fileSize = bytesToKbOrMb($bytesTotal);


							$bname=basename($saveToFile);
							$dname=dirname($saveToFile);

							#echo "<div align=lefy><pre>";
							#print_r($header);
							#echo "</div>";

							$ContentDisposition=trim(cut_str($header,'Content-Disposition:',"\n"));
							if ($bname == "attachment")
								{
									if ($ContentDisposition && strstr($ContentDisposition,"filename="))
										{
											$saveToFile_tmp=trim(cut_str($ContentDisposition."\n","filename=","\n"));
											list($saveToFile_tmp,$tmp)=explode(';',$saveToFile_tmp);
											$saveToFile=$saveToFile_tmp ? $dname.DIRECTORY_SEPARATOR.$saveToFile_tmp : $saveToFile=$dname.DIRECTORY_SEPARATOR.time().'_by_RapidKill';
										}
											else
										{
											$saveToFile=$dname.DIRECTORY_SEPARATOR.time().'_by_RapidKill';
										}
								}
									else
								{
									if ($ContentDisposition && strstr($ContentDisposition,"filename="))
										{
											$saveToFile_tmp=trim(cut_str($ContentDisposition."\n","filename=","\n"));
											list($saveToFile_tmp,$tmp)=explode(';',$saveToFile_tmp);
											$saveToFile = $saveToFile_tmp ? $dname.DIRECTORY_SEPARATOR.$saveToFile_tmp : $saveToFile;
										}
								}

							$saveToFile=str_replace('"','',$saveToFile);

							#echo "<b>$saveToFile</b><br>\n";

							list($saveToFile,$temp)=explode('?',$saveToFile);

							if (!$ResumeMode){
								$saveToFile = file_exists($saveToFile) ? dirname($saveToFile).DIRECTORY_SEPARATOR.time()."_".basename($saveToFile) : $saveToFile;
							}
							$fs = @fopen($saveToFile, "a");

							if(!$fs)
								{
									$lastError = "It is not possible to open the file ".basename($saveToFile)." in folder ".dirname($saveToFile)."<br>".
												"Wrong Installation, that the way for the retention is assigned correctly and in the script".
												"File couldn't be stored into this folder (try chmod the folder to 777).<br>".
												"<a href=javascript:location.reload(); style=\"color: #0000FF;\">Repeat</a>";
									return FALSE;
								}

							@flock($fs, LOCK_EX);

							$bytesSaved = fwrite($fs, $dopdata);

							if($bytesSaved > -1)
								{
									$bytesReceived += $bytesSaved;
								}
									else
								{
									echo "It is not possible to carry out a record in File".basename($saveToFile)."<br>";
									if (file_exists($saveToFile)) {unlink($saveToFile); }
									return FALSE;
								}
							@flock($fs, LOCK_UN);
							@fclose($fs);

							$chunkSize = GetChunkSize($bytesTotal);
							echo "File <b>".basename($saveToFile)."</b>, size <b>".$fileSize."</b>...<br>";
?>
<br>
<table cellspacing=0 cellpadding=0 style="FONT-FAMILY: Tahoma; FONT-SIZE: 11px;">
<tr>
<td width=100> </td>
<td width=300 nowrap>
	<div style='border:#BBBBBB 1px solid; width:300px; height:10px;'>
    		<div id=progress style='background-color:#000099; margin:1px; width:0%; height:8px;'>
    		</div>
	</div>
</td>
<td width=100> </td>
<tr>
<td align=right id=received width=100 nowrap>0 KB</td>
<td align=center id=percent width=300>0%</td>
<td align=left id=speed width=100 nowrap>0 KB/s</td>
</tr>
</table>
<script>
function pr(percent, received, speed, out)
{
document.getElementById("received").innerHTML = '<b>' + received + '</b>';
<?php
if ($title_update)
	{
?>
document.title='Downloading ' + percent + '% ['+orlink+']';
<?php
	}
?>
if (out)
	{
		document.getElementById("percent").innerHTML = '<b>' + percent +'%, '+out+'</b>';
	}
		else
	{
		document.getElementById("percent").innerHTML = '<b>' + percent +'%</b>';
	}
if(percent > 90){percent = percent - 1;}
document.getElementById("progress").style.width = percent + '%';

document.getElementById("speed").innerHTML = '<b>' + speed + ' KB/s</b>';
return true;
}

function minfo(str, field)
{
document.getElementById("mailPart." + field).innerHTML = str;
return true;
}
</script>
<br>
<?php
						}
				}
		}
			else
		{
			$page .= $data;
		}
}

if ($fp) {fclose($fp); }

if($saveToFile)
{
	if ($fs)
		{
			flock($fs, LOCK_UN);
			fclose($fs);
		}

	if($bytesReceived <= 0)
		{
			$lastError = "Wrong Link? Error Downloading File..<br><a href=\"javascript:history.back(-1);\">Go Back</a>";
			if (file_exists($saveToFile)) {unlink($saveToFile); }
			return FALSE;
		}

	if ($bytesTotal)
		{
			if ($bytesReceived < $bytesTotal)
				{
					$lastError = "Received <b>$bytesReceived</b> from <b>$bytesTotal</b><br>Connection Lost :-(<br>Incomplete file is removed<br><a href=\"javascript:history.back(-1);\">Go Back</a>";
					if (file_exists($saveToFile)) {unlink($saveToFile); }
					return FALSE;
				}
		}

	$fileSize = (!$bytesTotal) ? bytesToKbOrMb($bytesReceived) : bytesToKbOrMb($bytesTotal);

	return array("time"				=> sec2time(round($time)),
				"speed"				=> round($bytesTotal / 1024 / (getmicrotime() - $timeStart), 2),
				"received"			=> true,
				"size"				=> $fileSize,
				"bytesReceived"		=> $bytesReceived,
				"bytesTotal"		=> $bytesTotal,
				"file"				=> $saveToFile);
}
	else
{
    	return chunked($page);
}
}

shlomikalfa · December 21, 2008

Well.. it worked... but it appears like my script is still stuck on 20MBs

Any idea as to why would my script stop downloading at 20MB all the time ?!?

shlomikalfa · December 21, 2008

does anyone have a clue as to WHY ARE MY DOWNLOADS STOPPED IN THE MIDDLE ?!

shlomikalfa · December 22, 2008

Any idea guys ?!? anything i can check to or anything at all to try and have a look at ?!?

GIVE ME SOMETHING TO HOLD ON TO

shlomikalfa · December 23, 2008

Wow that's 2 days... on such a minor issue... any idea anyone ?!?

MadTechie · December 23, 2008

For download the big files (more than 8MB), you must used ob_flush()

because the function flush empty the Apache memory and not PHP memory.

And the max size of PHP memory is 8MB, but ob_flush is able to empty the

PHP memory.

header('Content-Type: application/force-download');

header ("Content-Length: " . filesize($file));

header ("Content-Disposition: attachment; filename=$theFileName");

$fd = fopen($file, "r");

while(!feof($fd))

{

echo fread($fd, 4096);

ob_flush();

}

shlomikalfa · December 23, 2008

@MadTechie

i have made my script output:

if ($bytesReceived>150000){echo "MemoryUsage: ".memory_get_usage()."<br />\r\n";}

and it never gets over 2.9 MB.....

Any other idea not memory related maybe ?!?

MadTechie · December 23, 2008

Could be a few things...

have you tried a direct download. (no proxy)

what about sessions timeing out .. (or cookies)

i wrote a scirpt that downloaded about 50 x 100-400Mb files without any real problems but that was direct

shlomikalfa · December 23, 2008

@MadTechie

-. Direct downloading [from my comp] works perfectly! (if that's what you meant)

-. I am not using proxy on host....

-. Cookies can time-out in a middle of a file download ?!?

I am trying to download a 200MB file by my host from rapidshare.... but it fails at 40MB [approx] which is my memory limit [on host] however... as i've already told you - it never shows anything bigger than 2.9MB on this echo:

if ($bytesReceived>150000){echo "MemoryUsage: ".memory_get_usage()."<br />\r\n";}

MadTechie · December 23, 2008

If you can give me a link of a large file i'll take a look, i don't have an account so it could be a pain..

in anycase try a simple script like this

<?php
echo download("http://file_to_download.zip", "localstorage.zip");

function download($filename, $write)
{
$chunksize = 1*(1024*1024); // how many bytes per chunk
$buffer = '';
$handle = fopen($filename, 'rb');
$writer = fopen($write, 'w');
if ($handle === false) return false;
while (!feof($handle))
{
	set_time_limit(0);
	fwrite($writer, fread($handle, $chunksize));
	ob_flush();
	flush();
}
$status = fclose($handle);
fclose($writer);
return $status;
} 

?>

shlomikalfa · December 23, 2008

nothing works

shlomikalfa · December 25, 2008

seems like the issue is in the php time limit in php.ini file or file size limit.

how can i make sure this is not the issue &/Or fix this issue if it's actually the issue ?!?

HELP PLEASE!

shlomikalfa · December 25, 2008

is this topic hidden or something ?

MadTechie · December 27, 2008

seems like the issue is in the php time limit in php.ini file or file size limit.

how can i make sure this is not the issue &/Or fix this issue if it's actually the issue ?!?

HELP PLEASE!

Why do you say thats the problem ?

is this topic hidden or something ?

Your not helping yourself by not giving ant useful infomation back,

IE

nothing works

Sign In

Memory limit issues when downloading a large file to host.

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Join the conversation

Important Information