Renaming files fails SOMETIMES

[Pelle]

Hello all. New guy here, so please go easy on me.

I am using plupload to upload files on a work website. The upload itself works flawlessly, but then I run into trouble. The website is hosted on a linux server with several other boxes mounted, whereof one file server and another server where the files are processed (let's call this the process-server). I.e. there are three separate file systems involved.

First copy: When the file upload is finished, the file(s) are copied from the linux server tmp-folder to the file server. This NEVER fails.

Second copy: Once on the file server, I proceed to copy the file from the file server to a watched folder the process-server. This failed about 40% of the time. I.e. it WORKS more often than not, but not 100% of the time. The only error I ever get is "No such file or directory in...". I do know for a fact, however, that this is wrong. The directory is there. I have done extensive logging if this procedure and watched it happen in front of my eyes. The directory NEVER exists when the upload starts, but is created before I get to the part where this error occurs.

Considering the possibility that the file is picked up from the watched folder before it is finished copying, I decided to FIRST copy the file to a NON-watched folder on the processor-server and then renaming it from the non-watched folder to the watched folder, since copying a file between file systems DOES take time. However, once on the correct file system, the rename:ing of the file is instant as the file is not physically moved. It's just file file system pointer that is changed.
This improved the success rate greatly, but not to 100%. Now I suffer this problem perhaps 10% of the time.

Below is the relevant code. What I do not understand is how this works most of the time but SOMETIMES fails. There is no consistency, this error occurs on large files and small files alike. This is all done on our intranet, so "slow internet connection" is not an issue. As you can see, the files are given a new name when copied to the file server. $filename contains a valid filename, without spaces, special characters and other weird characters.

Is it possible that the copy command returns true before the file is finished copying and allows the code continue to run? I thought I would not enter an if-statement block until the file was successfully copied and done. Is it thus possible that the rename executes and tries to rename the file before the copy is finished?
 

    if (copy("{$filePath}.part", $fileServerPath.$filename)) {  // File copied from webserver to file server. This NEVER fails.
            if (!copy($fileServerPath.$filename,$processServerPathTmp.$filename)) { // File copied from file server to process server tmp. This NEVER fails.
                error_log("Filehandler: Could not copy $filename from file server to process server tmp");  
            } else {
                if (!rename($processServerPathTmp.$filename,$processServerPath.$filename)) {  // File is moved from process server tmp to process server watched folder. This fails about 10%-20% of the time.
                    error_log("Filehandler: Could not move $filename from process server tmp to process server");
                }
            }
            return true;
    }

 

[requinix]

PHP won't be doing any weird things like returning from copy() when it's only partially done, so don't go chasing those white rabbits.

It's likely to be a filesystem issue. You talk about copying files between servers but presumably you mean that there's an NFS mount (or similar) on the local machine that is hosted by the remote machine? Could there be syncing problems?
Have you tried reproducing this problem without PHP? Copying files yourself from a shell, or even another language (eg, Python)?

[Pelle]

3 hours ago, requinix said:

PHP won't be doing any weird things like returning from copy() when it's only partially done, so don't go chasing those white rabbits.

It's likely to be a filesystem issue. You talk about copying files between servers but presumably you mean that there's an NFS mount (or similar) on the local machine that is hosted by the remote machine? Could there be syncing problems?
Have you tried reproducing this problem without PHP? Copying files yourself from a shell, or even another language (eg, Python)?

You were indeed correct. PHP does not do anything odd. And I was too cocky claiming that the "does not exist" error was incorrect.  If you get the error "does not exist", that's probably right. And yes, "between servers" means mounted drives... /mnt/fileserver, /mnt/processserver etc... as far as PHP is concerned, just different directories on the same machine.

This is what happened:

1) The script checks if all directories involved exists. They never do, so they are always created. I watched this happen, thus I "knew" that they existed.
Create fileserver/targetDirectory/, processServer/tempDirectory/, processServer/WatchedFolder/targetDirectory/ 

2) The files were uploaded to the web server. This took from a few seconds to half a minute, for large files

3) The files were copied from the web server to fileserver/targetDirectory/ (this always worked)

4) The files were copied from fileserver/targetDirectory/ to processServer/tempDirectory, to avoid copying to the watched folder. (This always worked)

5)  Then... sometimes failure when moving from processServer/tempDirectory/ to processServer/WatchedFolder/targetDirectory/

The failure occurred because the watching process scans processServer/WatchedFolder/ for files that have not changed in size the last 15 seconds. Remember that I created the processServer/WatchedFolder/targetDirectory before I even started the file upload! If the upload took "too long", the watching process noticed that the targetDirectory (just another file, after all) had not changed size the last 15 seconds and deleted it. So sometimes when I got to step 5 above, processServer/WatchedFolder/targetDirectory existed, sometimes it did not!

Many thanks for your input.

Edited September 17, 2021 by Pelle
typos and some clarifications

[gizmola]

Things like this, with interleaving batch processes also have a propensity to suffer from race conditions.  In your case it sounds like you have a form of a race condition, where separate processes interfered with your expectations of how things would work from a timing standpoint.

Sometimes the best tool for something is one that already exists.  

The entire process that copies from linux1 -> linux2 could be done using rsync, and its "--remove-source-files" option.  The command would be:

rsync --remove-source-files -options /path/to/src/ linux2host:/path/to/dest

This is not a criticism of what you built, but just pointing out that if the only reason you have a drive NFS mounted on linux1 is to facilitate this copy process, then rsyncing would decouple the servers, and also likely be a lot more efficient, allowing Linux1 and Linux2 to each do what they were designed for independently.   NFS has a good deal of network overhead intrinsic to it, and this copying process is likely producing a lot of network traffic that you don't need, just maintaining state and doing NFS locking.  NFS is great when you truly do need to share a filesystem across a number of different servers, but for a 1-1 mount like this, rsync would be better, and is also famous for its high level of performance in copying or syncing files.