Jump to content

About Large Data Volume Best Way To Deal With It


MiCR0

Recommended Posts

Unsure about the best way to go about doing this, I do my best to explain the problem.

I want to store tons like few mill text rows of data like.

 

KEY Bah bah abh 35 325 bah  234 2535 25352 35325 bah bah abh bah date time etc

*50 to *200 of the above

 

There will be around 20k to 50k outputs a day like above that need to be stored and like 1k to 5k of the outputs will be read. Because of the amount of data I am unsure MySql could handle it fast enough as well as the size would be crazy even if I do it a text or BLOB therefore I am thinking of doing it flat database.

 

Has anyone worked with this sort of data volume before and knows the best way to handle this. Problem with MySql is database Max size is 100MB on the server which I am using.

 

Link to comment
Share on other sites

hmmm, well if you could setup a cron job on your server, that exports your data as SQL - and emails it to you  that might work.  If however youll need that data to store and use continuely on the site, there will be a time, whether it be days or months, that youll need to upgrade your storage.  there are a few fantastic hosts out there that are very reliable, and have a fantastic data storage limit.  in the relm of GB ( and i did say reliable) - for only around 10-20$ a month.  I wont mention them here, but a quick google search shoudl be able to find you a few to start with.

 

Its really an issue of do you need that data on your server, or can you move it to your own harddrive daily, into another databse that you can still roam and query?

 

gdlk

Link to comment
Share on other sites

This thread is more than a year old. Please don't revive it unless you have something important to add.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.