Locking a file for passing data

[NotionCommotion]

For testing only, I wish to use a file as a simple queue.  Yea, I know I should be using a real queue, but I also wish to learn how to do it, and it is just for testing purposes only.  Is this correct?  Throughput will be low.  Thanks

 

Add to the buffer.

    public function saveTestData($json, $from, $to, $error=false)
    {
        $file='/var/www/test/buffer.json';
        $fp = fopen($file, 'r+');
        while (!flock($fp, LOCK_SH)) { usleep(1); }
        $existing = fread($fp, filesize($file));
        $new=$existing?json_decode($existing,true):[];
        $new[]=['from'=>$from,'to'=>$to, 'message'=>json_encode($json),'error'=>$error];
        ftruncate($fp, 0);
        fwrite($fp, json_encode($new));
        fflush($fp);
        flock($fp, LOCK_UN);
        fclose($fp);
    }

 

Get the buffer data and delete it.

function getTestData()
{
    $file='/var/www/test/buffer.json';
    $fp = fopen($file, 'r+');
    while (!flock($fp, LOCK_SH)) { usleep(1); }
    $existing = fread($fp, filesize($file));
    ftruncate($fp, 0);
    fwrite($fp, '');
    fflush($fp);
    flock($fp, LOCK_UN);
    fclose($fp);
    return $existing;
}

Edited March 10, 2017 by NotionCommotion

[NigelRel3]

I would have thought that you need to use LOCK_EX instead of LOCK_SH as in both cases you are writing to the file.  LOCK_SH allows shared access to the file.

[Jacques1]

The way you use flock() generally makes no sense. Your while loop indicates you expect non-blocking behavior, but you haven't set the LOCK_NB flag, so the function is blocking. What do you want? If you switch to non-blocking behavior and simply wait and try, you can end up waiting forever.

[NotionCommotion]

What do you want?

 

 Wait (forever if necessary) to be able to write to the file.

[Jacques1]

That wasn't the question.

 

flock() is blocking by default: If the file is already locked, then the process is suspended, added to a list and automatically woken up when the file is ready. This means there is no need for any waiting logic. The alternative is non-blocking behavior where the function immediately returns in any case. This implies you do have to write your own waiting logic, and if you get this wrong, a process may lose against other processes over and over again.

 

Since you appearently don't know what you want, I strongly recommend the default blocking behavior. So get rid of the while loop.

[NotionCommotion]

What confused me was this example from http://php.net/manual/en/function.flock.php.  If flock is blocking, what is the purpose of the if statement?

 

So, my writeSomething() will do the following?

1. Open a file.
2. Wait until it is available and instantly place a lock on it.
3. Truncate all data to zero (i.e. delete all content).
4. Add $txt to a buffer.
5. Flush it (i.e. write the buffer to the file).
6. Release the lock.
7. Close the file.

<?php

function writeSomething($txt) {
    #Example #1 flock() example
    $fp = fopen("/tmp/lock.txt", "r+");

    if (flock($fp, LOCK_EX)) {  // acquire an exclusive lock
        ftruncate($fp, 0);      // truncate file
        fwrite($fp, $txt);
        fflush($fp);            // flush output before releasing the lock
        flock($fp, LOCK_UN);    // release the lock
    } else {
        echo "Couldn't get the lock!";
    }

    fclose($fp);
}

writeSomething('hello');

[NotionCommotion]

Still questioning why if flock is blocking, why does the PHP docs use the IF statement.

 

This is what I now have.  Without even using the getTestData function, for some reason I am not appending the new records, but writing over the buffer.

 

function saveTestData($msg, $from=null, $to="null", $error=false) {
    $file='/var/www/test/buffer.json';
    $fp = fopen($file, "r+");
    if (flock($fp, LOCK_EX)) {
        $records=($filesize=filesize($file))?json_decode(fread($fp, $filesize),true):[];
        $records[]=['from'=>$from,'to'=>$to, 'message'=>json_encode($msg),'error'=>$error];
        ftruncate($fp, 0);
        fwrite($fp, json_encode($records));
        fflush($fp);
        flock($fp, LOCK_UN);
    } else {
        echo "Couldn't get the lock!";
    }
    fclose($fp);
}


function getTestData()
{
    $file='/var/www/test/buffer.json';
    $fp = fopen($file, "r+");
    if (flock($fp, LOCK_EX)) {
        $records=($filesize=filesize($file))?fread($fp, $filesize):'[]';
        ftruncate($fp, 0);
        fwrite($fp, '');
        fflush($fp);
        flock($fp, LOCK_UN);
    } else {
        echo "Couldn't get the lock!";
    }
    fclose($fp);
    return $records;
}

Edited March 10, 2017 by NotionCommotion

[kicken]

Still questioning why if flock is blocking, why does the PHP docs use the IF statement.

Because it's possible that flock fails for reasons other than the file already being locked. The way the default blocking behavior essentially works is it will only keep retrying as long as the error returned indicates that the file is already locked. If some other error occurs it'll just fail entirely.

 

The PHP manual doesn't really expand on all the possible errors. If you look at the underlying flock or fcntl function you can get an idea of what other kinds of situations might cause it to fail.

 

EBADF fd is not an open file descriptor

 

EBADF cmd is F_SETLK or F_SETLKW and the file descriptor open mode doesn't match with the type of lock requested.

 

EDEADLK It was detected that the specified F_SETLKW command would cause a deadlock.

 

EFAULT lock is outside your accessible address space.

 

EINTR cmd is F_SETLKW or F_OFD_SETLKW and the operation was interrupted by a signal; see signal(7).

 

EINTR cmd is F_GETLK, F_SETLK, F_OFD_GETLK, or F_OFD_SETLK, and the operation was interrupted by a signal before the lock was checked or acquired. Most likely when locking a remote file (e.g., locking over NFS), but can

sometimes happen locally.

 

ENOLCK Too many segment locks open, lock table is full, or a remote locking protocol failed (e.g., locking over NFS).

Edited March 10, 2017 by kicken

[Jacques1]

Common sense also helps. If, for example, the function receives a nonsense argument instead of a valid file resource, what is it supposed to do other than fail?

[NotionCommotion]

UGGG.  What am I doing wrong?  As seen, it only saves the last entry. Furthermore, if I run the file a second time, additional ^@^@^@^@ content is added (cat doesn't display it, but vi does and the content is obviously no longer valid json).

 

Thanks!

<?php
$file='myjson.json';

addRecord($file,'hello1');
addRecord($file,'hello2');
addRecord($file,'hello3');

echo(file_get_contents($file));

function addRecord($file,$msg) {
    $fp = fopen($file, "r+");
    if (flock($fp, LOCK_EX)) {
        $filesize=filesize($file);
        $json=$filesize?fread($fp, $filesize):'[]';
        $arr=json_decode($json);
        $newRow=(object)['name'=>$msg];
        $arr[]=$newRow;
        $new_json=json_encode($arr);
        ftruncate($fp, 0);
        fwrite($fp, $new_json);
        fflush($fp);
        flock($fp, LOCK_UN);
    }
    else{
        exit('Could not lock the file!!!');
    }
    fclose($fp);
}

Browser Output:

[{"name":"hello3"}]

[Michael@devserver public]$ vi  myjson.json
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@[{"name":"hello3"}]

 

[Jacques1]

You need to rewind the file pointer after truncation.

[NotionCommotion]

Thanks Jacques1,

 

Good news is I am no longer getting the empty space characters. Bad news is I am still not saving the earlier writes.

 

I went a little crazy with the rewinds (however, I also removed some places to check for change in effect).

 

I would expect rewind 1 isn't required because w+ automatically goes to the beginning.

 

I also don't think rewind 2 is required based on my reading of the documents.

 

Rewind 3 normally is required, but since I am truncating to 0, I wouldn't think it was.

 

Also looking at the documents, I didn't see any need of using ftruncate after the fwrite.

 

If starting with an empty myjson.json file, if I run the below script once, I get [{"name":"hello3"}].  If I run it twice, I get [{"name":"hello2"},{"name":"hello3"}], and if I run it again and again, I only get what it produced on the second run.

 

Is this expected?  My desired results are [{"name":"hello1"},{"name":"hello2"},{"name":"hello3"}] if I run it once, and [{"name":"hello1"},{"name":"hello2"},{"name":"hello3"},{"name":"hello1"},{"name":"hello2"},{"name":"hello3"}]if I run it twice, and so on.  How can this be accomplished?

 

Thanks

<?php
$file='myjson.json';


addRecord($file,'hello1');
addRecord($file,'hello2');
addRecord($file,'hello3');


echo(file_get_contents($file));


function addRecord($file,$msg) {
    $fp = fopen($file, "w+");
    //$fp = fopen($file, "c+");
    if (flock($fp, LOCK_EX)) {
        rewind($fp);    //rewind 1
        $filesize=filesize($file);
        $json=$filesize?fread($fp, $filesize):'[]';
        rewind($fp);    //rewind 2
        $arr=json_decode($json);
        $newRow=(object)['name'=>$msg];
        $arr[]=$newRow;
        $new_json=json_encode($arr);
        ftruncate($fp, 0);
        rewind($fp);    //rewind 3
        fwrite($fp, $new_json);
        rewind($fp);    //rewind 4
        fflush($fp);
        flock($fp, LOCK_UN);
    }
    else{
        exit('Could not lock the file!!!');
    }
    fclose($fp);
}

 

[Jacques1]

The “w+” mode truncates the entire file right from the beginning, so there's nothing to read. You need “r+”.

[NotionCommotion]

The “w+” mode truncates the entire file right from the beginning, so there's nothing to read. You need “r+”.

 

It would help if I learned to read...

 

Open for reading and writing; place the file pointer at the beginning of the file and truncate the file to zero length. If the file does not exist, attempt to create it.

[NotionCommotion]

No change in results.  Do you mind running this script, and see what you get?  Thanks

 

<?php
$file='myjson.json';


addRecord($file,'hello1');
addRecord($file,'hello2');
addRecord($file,'hello3');


echo(file_get_contents($file));


function addRecord($file,$msg) {
    $fp = fopen($file, "r+");
    if (flock($fp, LOCK_EX)) {
        //rewind($fp);    //rewind 1.  Not required since??? 'r+'    Open for reading and writing; place the file pointer at the beginning of the file
        $filesize=filesize($file);
        $json=$filesize?fread($fp, $filesize):'[]';
        rewind($fp);    //rewind 2.  Where is the pointer now?  http://php.net/manual/en/function.fread.php doesn't seem to say.
        $arr=json_decode($json);
        $newRow=(object)['name'=>$msg];
        $arr[]=$newRow;
        $new_json=json_encode($arr);
        ftruncate($fp, 0);
        rewind($fp);    //rewind 3  Per http://php.net/manual/en/function.ftruncate.php, the pointer isn't changed.
        fwrite($fp, $new_json);
        //rewind($fp);    //rewind 4.  Where is the pointer now?  http://php.net/manual/en/function.fwrite.php doesn't seem to say.
        fflush($fp);
        flock($fp, LOCK_UN);
    }
    else{
        exit('Could not lock the file!!!');
    }
    fclose($fp);
}

[kicken]

filesize

Note: The results of this function are cached. See clearstatcache() for more details.

Your $filesize is only going to be correct the first time the function is called. After that you'll get the cached value back. Either clear the cache or read the file in a way that doesn't depend on obtaining the file size.

[NotionCommotion]

Thanks kicken,

 

Well, don't know if it was the best answer as Jacques1's regarding +r (rookies mistake on my part) and rewind were pretty good also.  But that caching thing was pretty damn frustrating.

 

Ended up using only one rewind($fp) before the ftruncate statement.

 

Thanks all!

[kicken]

 

        //rewind($fp);    //rewind 1.  Not required since??? 'r+'    Open for reading and writing; place the file pointer at the beginning of the file

 

Correct, the rewind is not necessary, the file pointer begins at the start of the file in that mode.

 

 

        rewind($fp);    //rewind 2.  Where is the pointer now?  http://php.net/manual/en/function.fread.php doesn't seem to say.

 

fread advances the file pointer by the length of the data read, so it will be just past the last byte you read. In your case at the end of the file since you're reading everything.

 

       

 

        //rewind 4.  Where is the pointer now?  http://php.net/manual/en/function.fwrite.php doesn't seem to say.

 

Like fread, fwrite advances the pointer by the amount of data written, so the pointer would be just after the last byte written. In your case that would be the end of the file since you're starting with a blank file.

 

Of the two rewinds you have you only need one of them. ftruncate does not care about the file pointer's position so whether your rewind before or after it doesn't matter from an implementation point of view. I prefer to rewind after it as I feel it makes it more obvious what the code is doing.

 

While not necessary in your code, you might also want to look at fseek, this lets you move the file pointer arbitrarily. That is useful if you're dealing with file formats where you can calculate the offset to data (eg, fixed length records). You can also use it and ftell (or fstat) as an alternative to filesize if you have a pointer to a file but not path.

 

fseek($fp, 0, SEEK_END);
$filesize = ftell($fp);

Edited March 10, 2017 by kicken

[Jacques1]

You need to rewind the file pointer after truncation.

[NotionCommotion]

 

You need to rewind the file pointer after truncation.

 

 

Why after?  Per the documents http://php.net/manual/en/function.ftruncate.php

 

Note:

The file pointer is not changed.

 

[Jacques1]

When you're at the end of the original file content (after fread) and then remove that file content (with ftruncate), you obviously can't leave the file pointer at the original position and start writing. You need to go all the way back to position 0.

[NotionCommotion]

I guess technically I could rewind before ftruncate as ftruncate does not change the position and it will remain at zero, but will not do so, and will do as you recommend.  Thanks