Need advice processing a CSV file of 1000 lines

[OM2]

If I want to read in and process a CSV of a 1000 lines how easy is it to do without a database?

 

By process, I mean read in the data and do simple manipulations + compare each line with a line from another file, that maybe have 50 lines

 

I was just concerned that I might start hogging the CPU and memory of the server trying to process 1000 lines of data

It's VERY important that I can do WITHOUT a database

 

I'd only consider a database if it proves that not using will take too much CPU and memory

 

I've only just began considering making the application - can't give full details - as I've got to figure them out myself!

 

Any general advice would be great

 

Thanks

 

 

OM

[Psycho]

You don't need a database to process data, a database is primarily for storing the data. Even if you wanted to use a database, you would have to process it first anyway. But, if you have to compare values from one file to other files then a DB might make sense since it will be more efficient. To make a determination as to whether a DB would be more efficient or not will be dependent on how many files you need to compare and what "comparisons" you need to make.

[WebStyles]

don't worry, 1000 lines is nothing for a modern computer.

[OM2]

let's say i have 1000 lines - this probably won't every happen!

i will have one other single file that will contain maybe 100 lines

each one of the 1000 lines needs to be compared to the 100 lines in the second file

to be honest: i think i'd be ok just to leave out the comparison

 

the end purpose is to have a X number of users

hopefully this will be 1000 after a few months of launch of my service

 

does that change it in terms of required processing power?

would it be wiser to have a DB in that case?

 

actually... having said all of that, the CSV files do need to be reordered and stored as XML data

the XML data needs to be stored in alphabetical order - so some sorting does need to happen

 

let me know what u think

 

thanks

[WebStyles]

it all depends... are you expecting 1000 users per day (i.e. spread out through 10 to 12 hours), or at the same time?

 

Anyway, files with 1000+ lines are not that easy to maintain, and a database is very easy to set up. If I were you, I'd create a database. Safer, easier and faster.

[xyph]

Sorting can be done in PHP using array functions, but it's less efficient than using a database solution.

 

For scalability, I'd program it using a database. Can it be done quite quickly using flat files for ~1000 users? Yeah, no problem!

 

Keep in mind, comparing 1000 lines in one file with 100 lines in another could require up to 100,000 iterations of a loop structure. This isn't something you want to do every couple seconds. A solution to that could be to store an MD5 hash of each file, and the result of your comparison. As long as both files still match those md5 hashes, you can assume the result of the comparison will be the same, and you can simply grab the cached comparison result.

[OM2]

thanks guys for the replies

my first priority is to get a quick and dirty version ready that can be used

it's important that there is no setup required and that the code can run standlaone

the replies though have helped my thinking a lot

thanks