Looking for testers of a new php package

[timothymarois]

https://github.com/tmarois/Filebase

Looking for feedback, or issues/pull requests on github for a new package Ive released. 

It's also recognized on packagist for composer install, though it has no dependencies.

It's a Flat File database, all designed in files and setup that allows you to manually edit and use version control on your database. I wrote a lot of documentation to make it easy to understand.

I've got it up to 96% code coverage, but need some users to test it out and give some feedback. Designed the method APIs to be easy and intuitive, like a CRUD setup.

[Jacques1]

The library has fundamental design problems and is subject to both race conditions and data corruption.

 

For example, if two processes or threads read() the same file, increment a counter within the data and then write() their results, the second update overwrites the first one, and the counter is only incremented once. There's not even an error to indicate that something went wrong, the update is simply gone. Now imagine a real application which has to do far more complex updates all the time. When that's just a matter of luck, the user has a problem.

 

In addition to that, the write() operation itself doesn't even attempt to keep the files intact. It first truncates them and then starts to write the new content. If anything goes wrong now, the user is left with an empty or partially written file.

 

And of course the performance of constantly reading, decoding, sifting through, re-encoding and writing entire files is abysmal.

 

So, no, this is not a “MySQL alternative”, and you shouldn't market it as such. Realistically, this is an experimal library for small, unimporant files. Personally, I never quite understood the hype around “flat-file databases”, given that a) even the cheapest hoster has some kind of SQL database now and b) there's SQLite for embedded databases – but that's another story.

[ignace]

Looks good but it's worthless. It would have been much better if you would have used a validator component and a caching component.

 

Maybe wrote an extension for Doctrine or Eloquent for flat-file needs.

 

Then it would have been useful and probably would have a greater adoption rate. Nobody is interested in a package that when time comes to upgrade needs to rewrite all code.

 

Or that you have to learn a whole new way to validate or cache your data. Probably missing some things so that you have to hack in these things.

 

You can find great packages on Awesome PHP

[timothymarois]

The library has fundamental design problems and is subject to both race conditions and data corruption.

 

For example, if two processes or threads read() the same file, increment a counter within the data and then write() their results, the second update overwrites the first one, and the counter is only incremented once. There's not even an error to indicate that something went wrong, the update is simply gone. Now imagine a real application which has to do far more complex updates all the time. When that's just a matter of luck, the user has a problem.

 

In addition to that, the write() operation itself doesn't even attempt to keep the files intact. It first truncates them and then starts to write the new content. If anything goes wrong now, the user is left with an empty or partially written file.

 

And of course the performance of constantly reading, decoding, sifting through, re-encoding and writing entire files is abysmal.

 

So, no, this is not a “MySQL alternative”, and you shouldn't market it as such. Realistically, this is an experimal library for small, unimporant files. Personally, I never quite understood the hype around “flat-file databases”, given that a) even the cheapest hoster has some kind of SQL database now and b) there's SQLite for embedded databases – but that's another story.

 

Thank you for the feedback! Yes I am aware of the double writes which can lose data, another reason that makes flat file more difficult. But there are solutions that can be implemented. There is also the fact of writing challenges due to locking, which yes SQL is why statistical/transactional is best for. First I do like to say, SQLlite sometimes isnt the best option because it puts all of your data within one large file, that in itself has problems and overwhelming limitations. 

 

The idea of flat files is so you can manually edit documents without an admin or some type of interface, (like the growth of Yaml formats) and secondly it allows you to use version control on your database, this is increasingly becoming a bigger factor these days. I would say that so much content on a website is static and doesn't need to query the database every page load, but thats easily argued the user just needs to implement caching, so obviously not a selling point. 

 

You are also right, it is for small projects. I have an advertising optimization platform at one of my jobs, and I would never use flat files for that, MySQL has done a great job in securing of data and easily collecting reporting information. Ive had this for a while in small in house applications, and wanted to see if it was useful to anyone else so I went public. But I appreciate the feedback. 

[timothymarois]

Looks good but it's worthless. It would have been much better if you would have used a validator component and a caching component.

 

Maybe wrote an extension for Doctrine or Eloquent for flat-file needs.

 

Then it would have been useful and probably would have a greater adoption rate. Nobody is interested in a package that when time comes to upgrade needs to rewrite all code.

 

Or that you have to learn a whole new way to validate or cache your data. Probably missing some things so that you have to hack in these things.

 

You can find great packages on Awesome PHP

 

Thanks for the feedback! I didn't want to make it a component for a subset of use. It was designed as a standalone package, specifically for use on projects that I was working on at the time. You make a good point about the need to learn something new, but isn't that the same with anything you want to code with? Also upgrading needs a whole rewrite is not true if you stay within the versioning, that goes the same with any package you use, upgrading across major versions might need a whole rewrite depending on the depth, but thats still no reason why you don't choose a package. That happens everywhere. 

 

I needed a flat file solution, and there wasn't a good intuitive one available at the time, so I set out to do that. But I do appreciate the feedback, that was the purpose of this post. 

 

Thanks for the link to Awesome PHP, I will need to check out what they have there! 

[Jacques1]

But there are solutions that can be implemented.

 

Like what? Locking the entire file exclusively for an indefinite amount of time as soon as any process reads it? Using optimistic locks where the queries can constantly fail?

 

Since you also want to allow manual edits, you have an even bigger problem, because file operations outside of your interface completely ignore your locks. In other words: You have no idea who is writing what when.

 

 

 

First I do like to say, SQLlite sometimes isnt the best option because it puts all of your data within one large file, that in itself has problems and overwhelming limitations.

 

Like what? For the use case you're describing, SQLite is vastly superior in every aspect.

 

Flat files can be useful for something like configurations. But then I either want no library at all or just a few lines of code which correctly handle updates and don't corrupt my data.

[timothymarois]

Like what? Locking the entire file exclusively for an indefinite amount of time as soon as any process reads it? Using optimistic locks where the queries can constantly fail?

 

Since you also want to allow manual edits, you have an even bigger problem, because file operations outside of your interface completely ignore your locks. In other words: You have no idea who is writing what when.

 

 

 

 

Like what? For the use case you're describing, SQLite is vastly superior in every aspect.

 

Flat files can be useful for something like configurations. But then I either want no library at all or just a few lines of code which correctly handle updates and don't corrupt my data.

 

You do have a lot of good points I wont argue that, but lets say on settings/configurations, and let say that updates/edits do not happen on visitor level, but only in rare occasions where updates are made by one user at a time for the most part. Lets also say that the files are added into some type of version control to be shared and backed up. In these scenarios a package like this could work. I use it on some of my projects, and hugely for caching components. Yes its wrong of me to say its a complete SQL alternative, and I would never say use this to store statistical data. This is somewhat off topic but worth mentioning, Look at DynamoDB on AWS for a second, you pay for ever query you call, that itself to me is waste. Calling to a database in some other remote place is just an unnecessary hassle for static content. Now if you are storing thousands upon thousands of items, yeah don't use a flat file setup. But I am merely creating a better version of a key-value storage engine. Ive used MySQL for 10 years, its great, but web servers are getting faster, databases themselves are too much complexity for small simple projects. I'd rather work on my data out of the box instead of building a large relationship system, not to mention the changes to a database which you have to use migration systems to keep your team or anyone else using it up to date. At least with files, its a simple re-structure of the data.