Anyone had to return really large datasets from a RDBMS to Java, and overcome performance and memory issues, or used any third party software to overcome issues?
Any tips hints would be useful, we are trying to manipulate a large result set from SQL server.
Scenario: DB query returns 100Mb of data as a serialised object. This data is then analysed by our code.
Issues: Out of memory issue, we then have increased JVM memory spec, however we may need larger data sets. Performance ;o) Also there is siginificant overhead in the 100Mb as pure data is converted to an object.
Yes, I have an application which has to work with MySQL and Oracle. The JDBC driver from Oracle handled streaming ResultSets very well, the MySQL JDBC driver did not. I solved it by changing the resultset retrieve to ID's only and use a prepares statement to fetch each single records. A real nightmare, but the OOM problems were gone. On the performance side of things, it didn't even perform that bad, as the overhead of handling huge ResultSets was gone.