Can you upload a BIG file and split it into smaller pieces from the stream/buffe

[dsaba]

Background:

The server has limits on the sizes of files. I want to upload big files and then split them into smaller ones, thus the file sizes being within the limits of the server rules.

 

Question:

Is it possible to be able to upload a big file and save it by splitting it into multiple pieces, but without ever having a big file to start with?

 

Hypothesis:

I don't think its possible because PHP uses the HTTP protocol of uploading files via the html form method of the browse button. (<--- Feel free to correct me on the specifics) How would it be possible to upload a BIG file and split it into multiple pieces without ever having saving the BIG file first? Well, I suppose (but i'm not too knowledgeable on the specifics of the file upload process technicalities.. feel free to fill the gaps) that process would have to upload a little bit of the file at a time from the 'buffer' or 'stream' of data coming it (whatever you want to call it), each little bit it reads it can then save and move the pointer to the next bit of the file and continue saving it in small pieces. This way it would never have to upload the whole file before it starts the splitting process. Whether you can upload a file a little bit at a time with HTTP protocol or PHP I am not sure, and this is part of the point of this question.

That's my only idea of uploading a big file and splitting it into smaller pieces without having uploaded and saved the big file first.

 

Is my idea possible? If so, show me how. And/or give me your idea/solution.

I thought about actionscript and flash, but if it can't be done in PHP I don't see how flash could do it either since it communicates with HTTP protocols in file uploading as well....

 

 

 

-thanks

[The Little Guy]

you could store them in pieces.

 

Just read in the file (like opening it in notepad).

Cut the file up into smaller parts.

Save each part.

Then build a PHP page that can reassemble the files.

If it is an image you can display it on the browser, if it is something else, then you do something else with it.

[dsaba]

The goal is to take a BIG untouched file. Let PHP upload it and save it into multiple pieces of itself, without ever storing the original size of the file in a variable or file on the server.

So that won't work.

[trq]

I would say your better off looking into ftp for larger files, http really isn't designed for large file transfer.

[dsaba]

How would it be possible to upload a BIG file and split it into multiple pieces without ever having saving the BIG file first? Well, I suppose (but i'm not too knowledgeable on the specifics of the file upload process technicalities.. feel free to fill the gaps) that process would have to upload a little bit of the file at a time from the 'buffer' or 'stream' of data coming it (whatever you want to call it), each little bit it reads it can then save and move the pointer to the next bit of the file and continue saving it in small pieces. This way it would never have to upload the whole file before it starts the splitting process. Whether you can upload a file a little bit at a time with HTTP protocol or PHP I am not sure, and this is part of the point of this question.

 

 

I'd love for someone to address that.

[DeanWhitehouse]

people don't read all of long posts.

[The Little Guy]

The goal is to take a BIG untouched file. Let PHP upload it and save it into multiple pieces of itself, without ever storing the original size of the file in a variable or file on the server.

So that won't work.

 

That is what I explained in my previous post

[The Little Guy]

do they restrict it before you use this the "move_uploaded_file" function?

[dsaba]

That's beside the point little guy, but assume the file size restrictions are on variables and all files put on the server during any process.

You said you explained what I was asking earlier... Apparantly, I don't see how your method can avoid saving the size of the original file in a variable/file/temp. file.

Clarify how you solve this problem with your method.

[The Little Guy]

you would upload the file, such as bigFile.jpg

 

Then during upload, when it has been uploaded to: /temp/bigFile.jpg (or whatever directory php uploads it to before using the "move_uploaded_file" function), you then use readf function to read the file in as a string, and then take that string and break it into multiple parts, then save each part in its own file:

 

bigFile1.jpg.txt

bigFile2.jpg.txt

bigFile3.jpg.txt

 

Then you would save those files somewhere on your server.

Then you need to make a php file that can rebuild all 3 of those files and display it as one file (if you want to display it).

[roopurt18]

I'm assuming Little Guy's solution will not work because:

The server has limits on the sizes of files.

 

I'm guessing that if a user tries to create a file, even in the /var/tmp that is an upload, the server stops it from happening.

 

AFAIK, there is nothing you can do to get around this, otherwise the server admins wouldn't even have it in place.

 

This is one of those situations where you would probably need a Java or .NET component on your site that the user installs.  This component would be responsible for allowing the user to select a file to upload and if the file is larger than a certain size, the component would break it into chunks and upload them individually.

[dsaba]

I'm assuming Little Guy's solution will not work because:

The server has limits on the sizes of files.

 

I'm guessing that if a user tries to create a file, even in the /var/tmp that is an upload, the server stops it from happening.

 

AFAIK, there is nothing you can do to get around this, otherwise the server admins wouldn't even have it in place.

 

This is one of those situations where you would probably need a Java or .NET component on your site that the user installs.  This component would be responsible for allowing the user to select a file to upload and if the file is larger than a certain size, the component would break it into chunks and upload them individually.

 

Yes unfortunately, this is what I tried to explain in my very first post, which it took about 3 more repetitive posts for me to repeat myself for someone to get it. Only for you to tell me that it can't be done... I had already assumed this and talked about it (yep in my first post). If this is help I could have helped myself!

 

Rather, I would appreciate it if someone could tell me WHY or explain the upload process in (oh yeah i asked about this in my first post as well) of how it is impossible to read a little bit of the file at a time without storing it all in 1 variable/temp file. I've been waiting for someone to address this, and why you haven't I can only assume it is because you can't read or you are un-knowledgable about it.

 

 

I'm guessing that if a user tries to create a file, even in the /var/tmp that is an upload, the server stops it from happening.

My ask for help was never about you trying to guess or circumvent my server's restrictions, but about the concept I described in my first post.

[DeanWhitehouse]

Because (i think) when you upload, if you wish to split a large file is has to be stored in a cache for the script to split it, therefore you cannot upload a large file as the cache(buffer , whatever) cannot/will not store it.

 

 

[miracle_potential]

.htaccess thing you could change manually unless your talking gigabytes sort of stuff in which theres no real hope

[roopurt18]

Yes unfortunately, this is what I tried to explain in my very first post, which it took about 3 more repetitive posts for me to repeat myself for someone to get it.

No.  I got it on your very first post.  In fact the reason I responded to the thread was because people were constantly not understanding what you want to do and trying to stir you down the wrong path.

 

Only for you to tell me that it can't be done... I had already assumed this and talked about it (yep in my first post). If this is help I could have helped myself!

Help doesn't always have to be "This is how you solve the problem."  Sometimes help can be "You're assumptions are right and you're stuck."  BTW, I did tell you one method of solving the problem.  You can create an applet (either Java or .NET) that can run on the client's machine and break it apart before it ever reaches the server.

 

My ask for help was never about you trying to guess or circumvent my server's restrictions, but about the concept I described in my first post.

I have to make (educated) guesses because I don't know as much about your environment as either you or your server's administrators.  There is nothing about this topic I can say for absolute certain.  I'm also not trying to circumvent your server's restrictions.  You are trying to do that yourself by breaking a large file into multiple pieces.  I'm telling you what I might do in that same situation.

 

Rather, I would appreciate it if someone could tell me WHY or explain the upload process in (oh yeah i asked about this in my first post as well) of how it is impossible to read a little bit of the file at a time without storing it all in 1 variable/temp file. I've been waiting for someone to address this, and why you haven't I can only assume it is because you can't read or you are un-knowledgable about it.

Here again I am making an educated guess.  The file is being uploaded to a target URL handled by either apache or IIS most likely.  When apache / IIS realize they have an incoming file, they receive the entire file and only after they have received it do they bother to tell your PHP script that a file has been uploaded.  Thus, using PHP and your web server (apache or IIS), you have no way of detecting even where the temp file is until the upload is complete (which won't happen because your server disallows it).

 

Now we both know that there are sites that have upload progress meters so there must be a way of detecting and monitoring the temp file as it grows during the upload.  I have no experience with these but here are some more educated guesses.  First off, PHP has no built-in mechanism to accomplish this that I'm aware of, but there are probably PHP extensions that allow it.  However, I'm guessing you are unable to modify your PHP installation since you are likely not the admin of the machine.  I'm guessing that most other solutions to upload progress scripts are  uploading to perl or python scripts and are maybe not being handled by apache / IIS either (although I have no idea really).  You could probably play around with examples of those scripts to get an idea of what to do.

 

Now, let's say that you did have an idea of what to do.  Let's say you could monitor the upload and watch the file's size and once it reaches a certain barrier copy and move it.  You will be competing with whatever process is handling the upload in the first place.  This process might have the file opened exclusively, in which case there is nothing you can do.  Or it may not be exclusive, but then how can you be sure your process that "chunks" the file doesn't interrupt a read or write by the owning process?  You can't.  In all likelihood you will just corrupt the data.

 

Now you could write your own small web-server that can break files as they come in.  Depending on your experience level this might be very difficult and then you'd have to talk your web host into installing it on their machine. 

 

That is not likely to happen either.  So now we're back to  you creating a Java or .NET client component that can break the file on the client's end and send it to you in pieces.

 

Or upgrade your hosting plan.

 

Or get a new host.

[DeanWhitehouse]

If i am correct he can add the file using FTP, then can he use the PHP FTP functions?

 

http://w3schools.com/php/php_ref_ftp.asp