Hello,
I can't give you THE right thing to choose, but I'll try to give you some hints.
First of all : the DBMS. MySQL could be a good choice, but note that MySQL is not free for professional use. Moreover, don't ask MySQL to be as secure as Oracle ! Concerning performance, the front-end API will play an important role, but be aware that you would have more optimization choices with Oracle (though it would be long and hard to configure Oracle "optimally" for your own needs if you never worked in an Oracle environment). Basically, Oracle offers much more possibilities at the price of complexity. I will not speak much of SQL Server for I don't know it well, but from what I know, I think it might be a solution just between the first two ones, and might be a good compromise for you : finer tuning than MySQL with less complexity than Oracle.
Now, concerning your front-end. If you want a cross-platform application, choose Java. If performance is crucial : choose a C/C++ solution (JDBC, even optimized with autocommit=false, batch execution and row prefetching, is still much slower : 5 to 10 times from my experience compared to Oracle C++ API : OCCI). And if you want a nice interface (in a windows environment), choose C# (which I expect to be as slow as Java). You could also consider mixing these solutions, depending on your needs, or decide to be more homogeneous : it's up to you !
Note that what I said here is only my own opinion, but I hope I will have helped you a little.
Regards,
RBARAER