designing a social network - not sure on a couple of points

[spiderwell]

Hi All,

 

I have recently started a new role and its just me as developer, and the site is a social network, not a facebook rival, i might add .

However when it comes to the design, i was not having much luck researching online for a few areas, and hope that some of you might be able to point me in the right direction of design tutorials or just articles about such subjects.

 

Uploads

Obviously this is the one thing all social networks need, and I am wondering the best way to go about storage. I was planning on using the filesystem for storing physical files such as images/pdfs, rather than put in the database (text entered in browser will be stored in this way). Is there a best practicise for storage techniques, I am thinking of having a folder with the same name as the user_id for all the uploads to go into, and then if i resize any images, they will be generated on the fly and stored in a cache folder. Would this be the correct way to do this, as I cant imagine stuffing it all into 1 folder makes any sense. Perhaps i could take user folder to have folders for each upload type images/pdfs/doc etc.

 

Connections

Now this is the fun bit, I again have been researching but not really found any articles that give me the bones and meat of what to do here from a database table structure view or just concept design. My first thoughts are a table for friends which is just a look up of user ids essentially, that would be 3 columns, id PK col, user_id, friend_id , perhaps a date column, and confirm friend column too.

 

At this stage these are the only points that I am trying to work out, I am not looking for a coded answer, I like to be pointed in the right direction and then do the work myself, so anything that might be useful, please let me know!

 

Many Thanks in advance

 

Nick

[kicken]

Uploads

...

 

Splitting them up by UserID would work ok.  You may consider splitting up the user's directories also if you expect there may be a lot of users.  An alternative to splitting things by user id would be to just rename the file using either a random name, or maybe an auto-increment id of the row where the file details are stored. 

 

To split things up, one thing I do whenever I expect there may be a lot of files is divide them into sub directories based on the first couple letters of the name.  For example if I wanted to store the file 1928_somefile.jpg it would be stored into $UPLOADDIR/1/9/1928_somefile.jpg.

 

Connections

...

 

Basically a table like you mentioned is all you need to setup the connection.  You don't really need a separate auto-increment PK column though, just set your PK to be (userId, friendId). The combination should be unique and would be enough to identify the relationship in the table.

 

[spiderwell]

thanks for the feedback, i guess thats a composite pk? i think thats the phrase, where there is a PK from combination of 2 columns, so you cant end up being friends twice.

 

would your example of folders and files mean that i might end up with many folders/paths only leading to 1 file in a folder?

would that not lead to having twice as many folders as there are files?, and if so I am not sure what implication this would have on performance, if any.

[kicken]

would your example of folders and files mean that i might end up with many folders/paths only leading to 1 file in a folder?

would that not lead to having twice as many folders as there are files?, and if so I am not sure what implication this would have on performance, if any.

 

No, rather than ending up with one giant directory full of files (or directories) they get spread out over multiple directories.  If you just stuck it all into one directory and for some reason ever needed to go into that directory to do something you may have problems with things not handling a large number of files very well. 

 

If you want to have each user have their own folder, then you probably don't need to split the files up like that, just store them into the folder.  You might consider splitting the user's folders up that way though to prevent having a bunch of users folders in the same parent directory.  Such as maybe: Uploads/users/3/3456/somefile.jpg where 3456 is the user's ID#

 

 

[shlumph]

No, rather than ending up with one giant directory full of files (or directories) they get spread out over multiple directories. If you just stuck it all into one directory and for some reason ever needed to go into that directory to do something you may have problems with things not handling a large number of files very well.

To expand on this even more; there is a limitation on the number of "nodes" you can have in a certain directory. This number is much greater in ext4 filesystems than ext3, but still something you never want to reach.

[spiderwell]

ah yes, I think I came across that once before, about max number of files per folder, but it has slipped to the back of my mind!

[spiderwell]

this is what i have come up with, its does the splitting of up of a userid into each character and turns that into a path, and then adds a folder which is named from the user id

for instance user with id 12345 gets a path of /upload/1/2/3/4/5/12345.

define('UPLOADS_FOLDER', './upload/');


function make_user_upload_path($userid)
   {
       //create a path for the user for uploading
       if(strlen($userid) == 1)
       {
           if(!file_exists(UPLOADS_FOLDER . $userid)) {
               mkdir(UPLOADS_FOLDER . $userid,0777,true);
           }
           return UPLOADS_FOLDER . $userid . '/';//add trailing slash
       }
       else
       {
           $arrTemp = str_split($userid);
           $newPath = implode('/',$arrTemp);
           if(!file_exists(UPLOADS_FOLDER . $newPath . '/' . $userid)) {
               mkdir(UPLOADS_FOLDER . $newPath . '/' . $userid ,0777,true);
           }
           return UPLOADS_FOLDER . $newPath . '/' . $userid . '/';//add trailing slash
       }

   }

i think maybe i am splitting it up too much, maybe only split upto 3 or 4 characters would be enough?

Edited January 30, 2013 by spiderwell

[Christian F.]

You don't want to go too deep either, because that will greatly affect performance as well.

 

A while back I posted a link to a discussion that was on this specific topic; Storage design for a large number of files on a file systems. I'll see if I can't find it again, and give you the link.

In the meantime, I do recommend searching the forum for it as well; You might find it before I do.

 

Added:

Hah! I found it.

Edited January 30, 2013 by Christian F.

[spiderwell]

aha great stuff, thanks Christian, i shall peruse that link now

[spiderwell]

having looked at that article, and links to others in it, it seems that i do not want to go to deep, but i think i wont with the system i came up with, if i had 90 million users, the structure would be 8 folders deep. but i am a bit niave on things like this, perhaps that is too deep already?

[Christian F.]

For that many users, I'd look into a hashed directory structure. That way you'd ensure an even distribution, and you wouldn't have to create so many unnecessary directories. Nor levels.

[spiderwell]

well 90million is a figure i pulled out of thin air, as a top end example, personally if we get 1k users i'd be impressed, I shall do some more research on hashed directory structure, as it isnt an area i have any experience in

[kicken]

The hashed structure sounds like a pretty good general-purpose structure to me.  I may end up using that in the future.