Fast retrieving data from text file

[etrader]

I have a very simple system (few files) having simple articles (only title and a paragraph description) stored in single text files. I catch the data by

 

$data=file('article1.txt');
$title=$data[0];
$description=$data[1];

 

This simple system works perfectly, but the problem appears when increasing the number of files (e.g. more than 10,000 articles). This is a problem connected with the OS behavior for handling huge number of files. My first question: Is there theoretically a faster system (e.g. database-based) for retrieving data, comparing with retrieving from plain text file?

 

Second question: what can be an alternative for this very simple system (no search query or additional field)?

 

1. XML: It has the same problem if storing in different files.

2. Mysql: It is very advanced for this system.

3. SQLite:I am thinking of this, but it still have advanced structure of SQL.

4. DB Berkeley: I am not familiar with Oracle at all, and I need to install something new on my server. Is it really worth of trying?

5. Anything else?

 

In general, I just need fast reading the data. I have no idea how much the speed of these systems are different to determine which is worth of consideration. Thanks for sharing your idea.

[codefossa]

MySQL can suit basics as well as complex.  I would suggest using it as it makes life so much easier.

 

You shouldn't really use a flatfile system unless it's not going to be used by too many people at all.  The RAM usage when it is shoots up big time.

[xyph]

I'd suggest SQLite. No external service to run, native PHP support (php5), and quite quick.

 

Alternately, you could just use subfolders, perhaps 100 files per sub or a certain month's articles in a folder

[etrader]

Kira, I did not get how mysql uses lesser RAM comparing with file-based system.

 

xyph, Categorizing files in sub-folders is easy, and beneficial for the OS; but still I think it's a problem as the number of inodes is still limited.

[xyph]

That's why I suggested using SQLite.

 

I figured I'd provide an alternate filesystem solution as well. I don't quite understand why you're worried about an inode limit, unless you're on a shared server. With modern disks sizes, inode numbers are usually quite huge. I suppose if you're worried about having millions of articles at some point, that makes sense

[etrader]

No, I'm not on a shared server; but since this system uses very low amount of resources, I host many domains on the same server. I am thinking of shifting from ex3 to ex4; but I prefer to do so on a fresh server (maybe later).

 

SQLite seems to be an excellent choice, but I was unable to find some statistics comparing its speed with other systems. I just worry about RAM usage for concurrent visitors as SQLite does not have server (of course, my knowledge is very little about this).

[xyph]

Complex queries will get cached in more advanced database systems. If you can easily use a flat file to retrieve your data, I doubt you'll be running very complex queries.

 

I've never seen, but I've heard of 25+ GB single SQLite DB files working well on mid-range dedicated systems.

 

This may help

http://www.sqlite.org/whentouse.html