Jump to content

Function md5_file() Limits?


bsmither
 Share

Go to solution Solved by requinix,

Recommended Posts

PHP 7.4.2 is given 256M as memory_limit.

I am giving a 370MB file to md5_file(). I get a hash with no errors.

Is PHP loading the entire file into memory, all at once, to process it?

If so, is there then a practical limit to the size of the file without eventually causing an out-of-memory situation?

Link to comment
Share on other sites

Another experiment with a 8GiB file has these results:

PHP did not time out. Even though max_execution_time is set to 60 seconds, it took 66 seconds for the entire process to finish (this function and and a few milliseconds of additional work elsewhere).

The web server did time out waiting for PHP. Using NGINX where the default request timeout is 60 seconds, the result was a 504 Gateway Time-out.

PHP's current memory allocation jumped from 3MiB to 12MiB at the point where md5_file() executed.

Watching the drive activity light on the server box showed continuous activity for approximately 65 seconds. Followed by a brief blip for housekeeping.

So, I can surmise that PHP's md5_file() reads the file in chunks - perhaps 8-9MiB per chunk - but unknown if the file fits completely in memory, then that is what will happen.

 

Link to comment
Share on other sites

  • Solution

Here are some relevant facts:

1. max_execution_time has some nuances in exactly what counts towards the limit. For instance, on Linux systems it counts only the time PHP itself is working, and therefore will not count time for disk reads which are performed by the system.
2. The average hard drive can stream data at about 125 MB/s. An 8 GiB file would take about 8000/125 = 64 seconds to read in full.
3. PHP's memory usage is not a perfect correlation to the underlying source - especially given that md5_file() will be performing some amount of hashing work in addition to the literal file reads.
4. PHP claims memory in blocks. 8MB = some number of blocks taken * the memory used per block.

Do you want to know the exact truth of what PHP is doing, or would you like to continue investigating to see if you can discover it for yourself?

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.