Fatal Error

[batstanggt]

I have a script which is uploading very large files into blobs (<=800MB). Now i had the max memory size in the php.ini set to 150MB and while uploading it would stop and tell me that that I exhausted all the bytes (157xxxxx can remember the exact figure but im pretty sure it equalled 150MB). Any ways I experimentally set the max memory size to 800MB just to see if that would help and now its telling me Fatal error: Out of memory (allocated 524288) (tried to allocate 804524032 bytes)? What is the meaning of this and how can one rectify the issue? Any info would be tremendous!

-SB

[premiso]

You need to also change the upload_max_filesize and post_max_size size in the php.ini as well.

[gizmola]

Check a phpinfo() and insure that the memory setting matches what you have set.  If you have an 800mb file, you're going to need more than 800mb because overhead and any other variables require memory as well.  As for 800mb, that is 800 * 1024 * 1024 to get the actual byte number, which is what you see in these errors.

 

 

[PFMaBiSmAd]

A couple of points -

 

Using MB (with the B) on the end of a php.ini setting is invalid. Just use M

 

The memory error implies you are reading the uploaded file into a php variable. Do you need to do that or can you just move the uploaded file to its final location (which occurs through the operating system and does not consume php memory equal to the size of the file)?

[batstanggt]

Hey thanks for the quick responses. Premiso I already changed those settings to 800 and 900 with the post size being larger. PFM I changed the settings from 800MB to 800M and still the same deal. I have also reset the apache server after each modification so thats not the problem. Im curious why when I upped the max mem size the error reported less memory was allocated to handle the file than when the max mem size was 150? I apologize PFM I dont think I entirely understand your question. But I will try to answer best I can. I think I do need it in a Variable because I am inserting it into a mysql database. This script worked on smaller files as well so the script itself I dont think is the issue perhaps as much as the php.ini settings etc. I could be waaaaay off though too I know a fraction of a fraction of what many of you do. Thanks guys.

-SB

[batstanggt]

OH and I did try running phpinfo() and all my changes are displayed...

-SB

[PFMaBiSmAd]

I think I do need it in a Variable because I am inserting it into a mysql database.

 

You are planning on inserting an 800M Byte file(s) into a database? The default maximum size of data that you can transfer in one query (max_allowed_packet) is 1M bytes. Storing files in a database is not a good idea.

 

running phpinfo() and all my changes are displayed...

 

Php will display the setting (the MB), even if it is not a valid value.

 

Im curious why when I upped the max mem size the error reported less memory was allocated to handle the file than when the max mem size was 150?

 

Probably because the setting wasn't actually changed or it is being overrode some place else, such as in a .htaccess file or a local php.ini file or there simply isn't any more memory available to allocate.

 

The memory allocation error message implies that the file is being uploaded without error, but that you cannot read it into a variable due to the amount of available memory. You will either need to increase the max memory size, don't attempt to read the file into memory at all, or read the file in chunks using fopen/fread.

[premiso]

If all else fails and you have not done it yet an apache (or web server) restart may kick it in.

[batstanggt]

PFM yes. I am instering an 800MB file into a database or at least attempting to although it seems based upon your comment about max packet size thats probably not going to happen. The phpinfo() is giving me 800M which is what its supposed to be as I set it that way myself. I have already restarted the apache2 server and so I dont think that can be the cause of my problems and I dont know where to find or what an .htaccess file is? But in the minimal amount of reading Ive done it sounds like that pertains more to scripts that are uploading files to the directory or file system.  "You will either need to increase the max memory size(tried this afgter reading an article abiout how the max memory size in php needs to be double what youre trying to upload), don't attempt to read the file into memory at all (dont understand how to do this further explaination would be great), or read the file in chunks using fopen/fread (understand what this means just dont know how to go about doing it lol) ."

-SB

[PFMaBiSmAd]

Your current error message is?

[batstanggt]

Annnnd the current error message is ...... Fatal error: Out of memory (allocated 524288) (tried to allocate 804524032 bytes) in /var/www/newmymovies.php on line 36

-SB

[batstanggt]

OK this is really effing irritating me what the heck is causing the error to say that that i have .5mb (512kb) allocated when i have 1000MB allocated. This script worked on smaller files about 3mb and so the possibility of 512kb being set as the memory limit is assinine. Someone pleasse help!

-SB

[PFMaBiSmAd]

It would probably help at this point to post your code so that someone could see what it might be doing that is using up available memory and not releasing it at the point where you are reading in the uploaded file.

[DavidAM]

Annnnd the current error message is ...... Fatal error: Out of memory (allocated 524288) (tried to allocate 804524032 bytes) in /var/www/newmymovies.php on line 36

-SB

 

I think this is saying: "I currently (already) have 542,888 bytes allocated, and I tried to allocate 804,524,032 bytes. But there's not that much memory available." -- or something to that affect.

 

As PFMaBiSmAd said, there is a limit on the size of the packets when sending data to the database. If you insist on storing very large files in the database, you are going to have to look into reading chunks from the filesystem and writing chunks to the database. Reading is straightforward:

$fp = fopen("filename");
while ( !feof( $fp ) ) {
  $buffer = fread( $fp, 1024 );  // 1024 bytes or whatever size you want/can

  // send chunk to database
}
fclose($fp);

 

I'm not sure how to send chunks to a BLOB in the database, but I am pretty sure there is a way to do it.

 

However, I too, would stop and ask myself if it is necessary to save the file in the database. There may be a reason to do so, but if not, it would likely be best to store the file in the filesystem and store the path and filename in the database.

 

[PFMaBiSmAd]

To wedge a large file, piece by piece, into a column, you would need to concatenate each successive piece with the existing content already stored in the column - http://dev.mysql.com/doc/refman/5.5/en/string-functions.html#function_concat

 

You would need to perform the compliment of that process every time you try to retrieve the file.

[batstanggt]

OK new developments...When I upload a video under a mb it works when I upload a video 11mb it works then i try a 24 mb video and it doesnt work even though it says the file uploaded successfully..and the 800mb just gives me that anoying out memory error..does this info help at all?

-SB

[batstanggt]

Im also wondering based on my previous post if this could be a hardware issue...The site for testing purposes in being hosted on a crappy home made server (old desktop pc) ....so Im thinking could the "server" just not have enough RAM or something to upload a file as big as 800MB to it ?

-SB

 

[batstanggt]

NO input ?

-SB

[PFMaBiSmAd]

As already stated, the 800M byte file is being uploaded but at some point you are doing something that needs to allocate more memory than what is available.

 

Someone suggested that you post your code so that they/we/I/someone could see if your code is doing something that would consume memory and not make it available when needed. For all we know, the line number where the error is occurring at is after you have successfully read the file into a php variable, but you are doing something to it, like making a copy into another variable. If you have changed the code from the last time you posted the error message with the line number, repost the current error message, with the line number, that matches the code you post.

[batstanggt]

Hey Im not anywhere that I can access the code but as soon as I get home I will certainly post it. Thank you. Now I had been doing some of my own troubleshooting and noticed that there is a setting in Apache called LimitRequestBody and apparently this is as a default set to 512KB or 524288 bytes (possibly hence my Fatal error: Out of memory (allocated 524288) (tried to allocate 804524032 bytes) error. Now how does one configure apache ? I keep reading about directives but i dont know what these are, how to use them or even what they do really. Any insight would be great, and if anyone could check back later Ill have the code posted and I much appreciate your help in advance.

-SB 

[gizmola]

In the php.ini you need to look at these settings:

 

upload_max_filesize = 100M
post_max_size = 101M
max_input_time=  300

 

In this example, I'm suggesting you start with a setting of 100m and test uploads of files around the 95m range.  Once you have success you can start upping the limits, using larger files and rechecking.  post_max_size must always be a bit larger than upload_max_filesize, and max_input_time needs to long enough (in seconds) for the upload to complete.  Depending on your upstream, for an 800m file, this may need to be significantly increased.

 

The question of LimitRequestBody is based on your apache configuration.  If there is a config file being loaded that has imposed this restriction then you need to find that and adjust it upwards.  Check your httpd.conf and any directories or files that are being included.

 

If you have mod_info setup on your server, you can check the current settings and determine where configurations are set, but I doubt that LimitRequestBody has been configured.  If you're unclear on mod_info, do a search of the httpd.conf for it.  You might need to tweak it in order to access the page from your IP, even if it is turned on.  Often it is available but commented out.  When enabled you can open http://yourserver/server-info, search that for LimitRequestBody and you'll probably find that it's:

 

    1036:   LimitRequestBody 0
        : 

 

Which in this case is showing that a particular vhost on my server has this setting configured to be 0 (no limit)  and that this setting was made in the httpd.conf file at a particular line.  So another way to be sure this is turned of would be to go to the virtualhost section for the site in question and add it to the bottom:

 

LimitRequestBody 0

 

If you go through the server-info and there is no explicit setting of the variable, it is at the default of 0.

 

[PFMaBiSmAd]

Interestingly, I just discovered that using mysqli prepared statements would (supposedly, if it works) let you send blob data in packets by specifying the 'b' type in the bind_param statement -

 

b corresponding variable is a blob and will be sent in packets

 

Retrieving that data in packets/chunks would apparently be up to you.

[batstanggt]

Sorry for the delay guys...

the error

Fatal error: Out of memory (allocated 524288) (tried to allocate 804524032 bytes) in /var/www/movieupload.php on line 36.

 

the script

movieupload.php

<?php
ini_set("display_errors", "1");error_reporting(-1);  
$db_host = 'localhost'; // don't forget to change 
$db_user = 'root'; 
$db_pwd = 'dbpwd';
$database = 'dbname';
$table = 'movies';
// use the same name as SQL table
$password = '123';
// simple upload restriction,
// to disallow uploading to everyone
if (!mysql_connect($db_host, $db_user, $db_pwd))    
die("Can't connect to database");
if (!mysql_select_db($database))    
die("Can't select database");
// This function makes usage of
// $_GET, $_POST, etc... variables
// completly safe in SQL queries
function sql_safe($s){    
if (get_magic_quotes_gpc())        $s = stripslashes($s);    
return mysql_real_escape_string($s);}
// If user pressed submit in one of the forms
if ($_SERVER['REQUEST_METHOD'] == 'POST'){    
// cleaning title field    
$title = trim(sql_safe($_POST['title']));
if ($title == '') 
// if title is not set        
$title = '(empty title)';
// use (empty title) string    
if ($_POST['password'] != $password)// cheking passwors        
$msg = 'Error: wrong upload password';
else    {        
if (isset($_FILES['movie']))        {            @list(, , $mtype, ) = getimagesize($_FILES['movie']['tmp_name']);            
// Get file type.
// We use @ to omit errors
{                $data = file_get_contents($_FILES['movie']['tmp_name']);
$data = mysql_real_escape_string($data);
// Preparing data to be used in MySQL query                
mysql_query("INSERT INTO {$table}                                SET title='$title', movie='$data'");
$msg = 'Success: movie uploaded';
}
}
elseif (isset($_GET['title']))      // isset(..title) needed
$msg = 'Error: file not loaded';
// to make sure we've using
// upload form, not form
// for deletion
if (isset($_POST['del'])) 
// If used selected some photo to delete        
{                         
// in 'uploaded images form'
$id = intval($_POST['del']);
mysql_query("DELETE FROM {$table} WHERE id=$id");
$msg = 'Movie deleted';
}    
}
}
elseif (isset($_GET['show'])){    $id = intval($_GET['show']);
$result = mysql_query("SELECT id, UNIX_TIMESTAMP(movie_time), movie FROM {$table}                            WHERE id=$id LIMIT 1");
if (mysql_num_rows($result) == 0)        die('no song');
    list($ext, $movie_time, $data) = mysql_fetch_row($result);    $send_304 = false;    if (php_sapi_name() == 'apache') {
// if our web server is apache
// we get check HTTP        
// If-Modified-Since header
// and do not send image        
// if there is a cached version
$ar = apache_request_headers();
if (isset($ar['If-Modified-Since']) && 
// If-Modified-Since should exists            
($ar['If-Modified-Since'] != '') && 
// not empty            
(strtotime($ar['If-Modified-Since']) >= $movie_time)) 
// and grater than            
$send_304 = true;                                     
// movie_time    
}    
if ($send_304)    {        
// Sending 304 response to browser        
// "Browser, your cached version of image is OK        
// we're not sending anything new to you"        
header('Last-Modified: '.gmdate('D, d M Y H:i:s', $ts).' GMT', true, 304);        
exit(); 
// bye-bye    
}    
// outputing Last-Modified header    
header('Last-Modified: '.gmdate('D, d M Y H:i:s', $movie_time).' GMT',            true, 200);    // Set expiration time +1 year    
// We do not have any photo re-uploading  
// so, browser may cache this photo for quite a long time    
header('Expires: '.gmdate('D, d M Y H:i:s',  $movie_time + 86400*365).' GMT',            true, 200);    
// outputing HTTP headers    
header('Content-Length: '.strlen($data));    
header("Content-type: video/mpeg");    
// outputing book    
echo $data;    
exit();
}
?>
<head>
<title>MySQL Blob Image Gallery Example</title>
<link rel="stylesheet" type="text/css" href="test.css"/>
</head>
<html>
<body>
<?php
if (isset($msg)) 
// this is special section for
// outputing message
{
?>
    <p style="font-weight: bold;"><?=$msg?><br><a href="<?=$_SERVER['PHP_SELF']?>">reload page</a><!-- I've added reloading link, because     refreshing POST queries is not good idea --></p>
    <?php
    }
?>
    <h1>Movies</h1>
    <h2>Uploaded Movies:</h2>
    <form action="<? $_SERVER['PHP_SELF'] ?>" method="post">
    <!-- This form is used for image deletion -->
    <?php
    $result = mysql_query("SELECT id, movie_time, title FROM {$table} ORDER BY id DESC");
if (mysql_num_rows($result) == 0) 
// table is empty    
echo '<ul><li>No Movies Loaded</li></ul>';
else{    
echo '<ul>';    
while(list($id, $movie_time, $title) = mysql_fetch_row($result))    {        
// outputing list        
echo "<li><input type='radio' name='del' value='{$id}'>";        
echo "<a href='{$_SERVER['PHP_SELF']}?show={$id}'>{$title}</a> – ";        
echo "<small>{$movie_time}</small></li>";    
}    
echo '</ul>';    
echo '<label for="password">Password:</label><br>';    
echo '<input type="password" name="password" id="password"><br><br>';    
echo '<input type="submit" value="Delete selected">';
}
?>
    <html>
    <body>
    </form><h2>Upload New Movie:</h2><form action="<? $_SERVER['PHP_SELF'] ?>" method="POST" enctype="multipart/form-data">
    <INPUT TYPE="hidden" NAME="MAX_FILE_SIZE" VALUE="1000000000">
    <label for="title">Title:</label><br><input type="text" name="title" id="title" size="64"><br>
    <br>
    <label for="movie">Movie:</label><br><input type="file" name="movie" id="movie"><br>
    <br>
    <label for="password">Password:</label><br><input type="password" name="password" id="password"><br>
    <br><input type="submit" value="upload"></form>
    </body>
    </html> 

Thanks guys.

-SB

 

[batstanggt]

Anybody have any insight?

-SB

[batstanggt]

So now since ive beebn screwing around with the pho.ini etc none of my site works. I would really appreciate some help or im screwed since now the whole site is effed.

-SB