hackalive Posted June 9, 2010 Share Posted June 9, 2010 Thanks to ignace and MrAdam I have managed to complete my web crawler script for a seach engine. Considering the scale this may grow to I need a clean and efficient DB design, so I am asking how you guys would do it. It would need somewhere for all the keywords and be able to link them to certain sites One table (tbl_site) might work like this |id|url|title|description Just looking for your guys opinion and how you think this DB could/may/should work Thanks in advance Quote Link to comment Share on other sites More sharing options...
hackalive Posted June 10, 2010 Author Share Posted June 10, 2010 bump Quote Link to comment Share on other sites More sharing options...
hackalive Posted June 12, 2010 Author Share Posted June 12, 2010 bump Quote Link to comment Share on other sites More sharing options...
ignace Posted June 12, 2010 Share Posted June 12, 2010 I don't think SE use a traditional DB like MySQL maybe something like NoSQL? Quote Link to comment Share on other sites More sharing options...
hackalive Posted June 12, 2010 Author Share Posted June 12, 2010 okay, i'll take a look thanks ignace Quote Link to comment Share on other sites More sharing options...
Mchl Posted June 12, 2010 Share Posted June 12, 2010 Google for example has developed it's own storage engine which is used in many of their services: http://en.wikipedia.org/wiki/BigTable Quote Link to comment Share on other sites More sharing options...
hackalive Posted June 12, 2010 Author Share Posted June 12, 2010 okay so if I was to use cassandra (for example) how would I organise the tables do you guys think? Thanks for the help so far Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.