Speed of large arrays

[drifter]

I have a very large data file that I need to parse every day - somewhere around 5MB.

Each line is tab delimited, so I must explode, then I use MYSQL to insert or update into a database.

I know that realistically, probably only 5% of the lines change each day, so I was wondering if I could make this faster by storing yesterdays file in and then using arrays...

Basically, I have never used much speed testing with large arrays, so I have a 5MB file. I read it into an array and loop through to handle it.

Would it be faster to just update the DB for every line, or to search another very large array that contains yesterdays data, and only process if there is a change?

To make it even more simple... how fast is it if I use in_array with a 5mb array? Faster then just exploding each line in the array and write it to a database?

[Barand]

The fastest way is probably LOAD DATA INFILE

http://dev.mysql.com/doc/refman/4.1/en/load-data.html

[Psycho]

The factors that determine this would be determined by the specs of the machine that is runing the code and how the code is written. If it is becomming an issue, then it probably wouldn't take long to rewrite the code to use arrays. Then you could see for yourself how long it would take.

If the current process is not taking too long, I'd just pushing everything up to the database. However, one other thing you could do which may or may not be possible would to use a "last update" trigger. This would require that you can get the time/date stamp that the record was last updated included in the data file. Then you would only need to add/update those records.