i am doing a search engine project. that need a big database access. my extimate that database size may be 100gb. which database i hvae to use. my friends told me that you have to develop your own engine for that. For that what i have to do. i have no idea. please help me. i want to develop that from open source database like MySQL, SQLite.. is it right? if so tell me where can i get the sourcfe code with documentation. i know that source code is available in MySQL.org. i need the engine[access layer] code alone..
For implementing that which is the best language. and which is more efficiency.. i want to develop that in Linux platform. is it correct choice? if not which os is best for database access..
i am waiting for the reply of my seniors..............
I would not consider writing a database engine for this kind of task. The cost would be WAY too high to be able to justify for a single project, and it truly isn't necessary.
There are a number of commercial products that you could use. DB2, Sybase, and Oracle jump right to my mind. While you have to buy the engine once you go to production, all of them have developer versions that are either free or very low cost. In the long term, this would be the cheapest solution because you'd have a stable database engine, you can get good support, you can hire programmers as needed, and you'll save millions of dollars in the hunt to find investors when you try to sell the engine (compared to selling a similar product with a proprietary database engine).
I think you could probably get a database this size working with MySQL too. I've never done it myself, but I know several people that have done it.
I'm just curious, but what kind of web search engine do you envision that only needs 100 Gb of storage? Are you looking to create a product to provide for single web sites like http://www.dbforums.com to search just their own content? That market is getting rather crowded, and 100Gb wouldn't even do this site by itself if you include the archives.