I currently am using MySQL with MyISAM and InnoDB tables. The database is used for a large set of message boards (30,000 boards and 10 million posts).

The problem is a rare slower query takes a few seconds to complete, while it runs other queries get stuck behind and MySQL reaches its max connection limit. I have optimized the database design, the queries, and mysql's configuration.

So now I am looking for an alternative:
Needs to have something other than table level locks. InnoDB did this but uses more RAM than MyISAM.
Needs to live with limited RAM (1GB).
Should be under $1000 a year for a license.

Speed of database engine also depends on operating system.
I am using MySQL and i have cca 4GB database without speed problems.
If you are using Windows try to migrate to Unix like OS. Be familiar with it and tune it.