Results 1 to 2 of 2
  1. #1
    Join Date
    Jan 2003

    Post Unanswered: Large results sets in java

    Anyone had to return really large datasets from a RDBMS to Java, and overcome performance and memory issues, or used any third party software to overcome issues?

    Any tips hints would be useful, we are trying to manipulate a large result set from SQL server.

    Scenario: DB query returns 100Mb of data as a serialised object. This data is then analysed by our code.

    Issues: Out of memory issue, we then have increased JVM memory spec, however we may need larger data sets. Performance ;o) Also there is siginificant overhead in the 100Mb as pure data is converted to an object.

  2. #2
    Join Date
    Aug 2003
    Yes, I have an application which has to work with MySQL and Oracle. The JDBC driver from Oracle handled streaming ResultSets very well, the MySQL JDBC driver did not. I solved it by changing the resultset retrieve to ID's only and use a prepares statement to fetch each single records. A real nightmare, but the OOM problems were gone. On the performance side of things, it didn't even perform that bad, as the overhead of handling huge ResultSets was gone.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts