nonexistentera Posted October 20, 2008 Share Posted October 20, 2008 I have a very large table, ~100 million rows, and many times when I connect to it, the server shutsdown from exceeding CPU quota(shared host). Is there anyway to do this better. I still need to search through it too, so Im wondering If I could do a file with all the data, like an XML file. Im not sure, and I REALLy don't want to over do it. Thanks in advance for any options. Link to comment Share on other sites More sharing options...
corbin Posted October 20, 2008 Share Posted October 20, 2008 You have a table with 100m rows and you're on shared hosting? Just of curiosity, what kind of application needs 100 million rows and doesn't warrant dedicated hosting? Link to comment Share on other sites More sharing options...
dropfaith Posted October 20, 2008 Share Posted October 20, 2008 100 million rows on a shared host is a horrible idea imo buy a dedicated host for this project is step one Link to comment Share on other sites More sharing options...
meman1188 Posted October 20, 2008 Share Posted October 20, 2008 100 Million! thats insane, i'm surprised your host hasn't kicked you out yet. The only thing that comes to mind is the Zend implementation of Lucene. (http://framework.zend.com/manual/en/zend.search.lucene.html) This would allow you to keep the search capabilities but you would be moving the burden to the filesystem. Just an option to look at, but you really need to consider a dedicated host if your requirements are that high. Also I will second the above, what are you doing that has 100 million rows? Link to comment Share on other sites More sharing options...
nonexistentera Posted October 21, 2008 Author Share Posted October 21, 2008 It's more of a project for school Link to comment Share on other sites More sharing options...
corbin Posted October 21, 2008 Share Posted October 21, 2008 You have a project for school on shared hosting? Strange. Just wondering, what exactly is the nature of the data you are storing? The more info you give us, the more we can help. Link to comment Share on other sites More sharing options...
dropfaith Posted October 21, 2008 Share Posted October 21, 2008 100 million rows is still unexplained i cant imagine that many rows. for anything short of myspace sized websites and this being a school project confuses me even more Link to comment Share on other sites More sharing options...
PFMaBiSmAd Posted October 21, 2008 Share Posted October 21, 2008 Almost all the threads started by nonexistentera deal with making a table of all possible combinations, including the md5() of the values. I would say this has more to do with making an md5 reverse look up table or making a brute force password table. Link to comment Share on other sites More sharing options...
corbin Posted October 21, 2008 Share Posted October 21, 2008 Oh fun!!!! The storage space for any kind of useful table would be in the TBs, maybe even PB. If this is what he's trying to do, <3 script kiddies. Link to comment Share on other sites More sharing options...
nonexistentera Posted October 21, 2008 Author Share Posted October 21, 2008 @PFMaBiSmAd Good job you guessed it pretty well. And yeah I would be massive. I think I did pretty well though. I was at 20 million and only took up around 800mb's. And I do understand that it would be huge. Thats why I was wondering if I could just put it in a file. It could be in an Array or some nice and simple form. BTW I have unlimited space and bandwidth(within reason) Link to comment Share on other sites More sharing options...
DarkWater Posted October 21, 2008 Share Posted October 21, 2008 A file would probably be even larger....I mean, MySQL uses files too (in its actual implementation, you just don't see it), but it manages them in a more usable way than you could probably do with normal file storage. Link to comment Share on other sites More sharing options...
AndyB Posted October 21, 2008 Share Posted October 21, 2008 Almost all the threads started by nonexistentera deal with making a table of all possible combinations, including the md5() of the values. I would say this has more to do with making an md5 reverse look up table or making a brute force password table. @PFMaBiSmAd Good job you guessed it pretty well. Given the guess and the admission by the OP, it strikes me that this isn't the sort of thing where phpfreaks should help. Unless the OP can make a convincing case for allowing it to continue, this thread is locked. Link to comment Share on other sites More sharing options...
Recommended Posts