Large results sets in java [Archive]

dvmistry

01-10-03, 13:40

Anyone had to return really large datasets from a RDBMS to Java, and overcome performance and memory issues, or used any third party software to overcome issues?

Any tips hints would be useful, we are trying to manipulate a large result set from SQL server.

Scenario: DB query returns 100Mb of data as a serialised object. This data is then analysed by our code.

Issues: Out of memory issue, we then have increased JVM memory spec, however we may need larger data sets. Performance ;o) Also there is siginificant overhead in the 100Mb as pure data is converted to an object.

DaCoTa

09-30-03, 11:16

Yes, I have an application which has to work with MySQL and Oracle. The JDBC driver from Oracle handled streaming ResultSets very well, the MySQL JDBC driver did not. I solved it by changing the resultset retrieve to ID's only and use a prepares statement to fetch each single records. A real nightmare, but the OOM problems were gone. On the performance side of things, it didn't even perform that bad, as the overhead of handling huge ResultSets was gone.