I would not consider writing a database engine for this kind of task. The cost would be
WAY too high to be able to justify for a single project, and it truly isn't necessary.
There are a number of commercial products that you could use. DB2, Sybase, and Oracle jump right to my mind. While you have to buy the engine once you go to production, all of them have developer versions that are either free or very low cost. In the long term, this would be the cheapest solution because you'd have a stable database engine, you can get good support, you can hire programmers as needed, and you'll save millions of dollars in the hunt to find investors when you try to sell the engine (compared to selling a similar product with a proprietary database engine).
I think you could probably get a database this size working with MySQL too. I've never done it myself, but I know several people that have done it.
I'm just curious, but what kind of web search engine do you envision that only needs 100 Gb of storage? Are you looking to create a product to provide for single web sites like
http://www.dbforums.com to search just their own content? That market is getting rather crowded, and 100Gb wouldn't even do this site by itself if you include the archives.
-PatP