Downloading a "huge" file from another server

[btray77]

Hi, what would you suggest for downloading a HUGE file from another server?  I need to download a large CSV file, about 1.6 GB and insert it into the database.  right now I'm FTPing the file over, and using MySQL's LOAD DATA INFILE but I need to automate the downloading of the file.

 

Thanks

 

-Brad

[ignace]

Use CRON to automate the download. To increase performance which I seriously recommend with such size can be found at http://dev.mysql.com/doc/refman/5.1/en/insert-speed.html

 

I also strongly advice to use some other method then just download and afterwards 'load date infile' as everytime the CSV gets updated you need to download the whole 1.6 GB again and it's size will only increase and what about the rows you have already in your database when you re-download the whole 1.6 GB? If for example only 500 lines were added instead store the position of your pointer in the big file and next time you need to download read from the position of your pointer to the end of the line, store those lines in a local file and use that as your next 'load data infile'

[ignace]

You may also want to check your operating system and the maximum filesize it can handle as you may find yourself one day not adding any data while on the other end still data is being added. The same goes for your total disk space available.

 

Windows: http://technet.microsoft.com/en-us/library/dd316417.aspx

Linux: http://doh.vorax.org/docs/suse91/apas04.html

[btray77]

@ ingace

 

Thanks for the information.  To do the insert i'm using what I learned from this tutorial..  http://www.softwareprojects.com/resources/programming/t-how-to-use-mysql-fast-load-data-for-updates-1753.html  Data insertion is very fast, with this.

 

I just need a way to get the file "fast" programicaly in php or possibly perl VS having to wait a predetermined amount of time with a cron.  The updates are not always at the same time. 

 

Thanks

-Brad