Jump to content

file system performance; how many files in one dir


fatkatie

Recommended Posts

I understand structure for human needs but does a unix file system need it?

 

I'm accumulating a lot of image files.  After I  mangle the file name, the raw image files are being dumped in a single directory while all the image file's meta data is being stored in a database.  Does a unix file system prefer structure, lots of little directories, or does it deal just fine with many files in one.

 

All 'find that file' processes begin with the database.

 

 

Thank you.

Link to comment
Share on other sites

Depends on the filesystem. ext4 (most common?) doesn't have a limit per directory. However directories themselves take space to store a list of their contents, and unless the system sorts directory entries (?) looking for a file requires scanning the directory list for it.

 

It might be antiquated now but if you know the filename from another source then I'd partition. I tend to generate hex filenames and take the first two bytes' worth as two subdirectories.

/01/23/0123456789abcdef.ext
Assuming an even distribution, by the time you've filled each second-level directory with 256 files you've stored 16M total. More than enough.
Link to comment
Share on other sites

I was thinking about hashes and cache at the time and thinking size made NO difference.  But, then, I don't know.  And in the end size does always seem to matter.  Your method looks like a good way to 'distribute'.  I'll use it.  Looking at a directory with so many files has me on edge... like something is going to blow.  Thanks for tip.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.