Results 1 to 5 of 5
  1. #1
    Join Date
    Oct 2009
    Posts
    2

    Unanswered: Need help for migarating huge data into postgresql

    Hi,

    I have 40 millions of data to migrate into a single table of Postgresql database. I need to improve the performance of postgres database to import such a huge data. I need you people to tune the database and tables to import these data with high performance.

    Also, I need any effective migration tools which supports postgres.

    Help regarding this will be highly appreciated.

    Thanks
    Bala

  2. #2
    Join Date
    Aug 2009
    Location
    Olympia, WA
    Posts
    337
    You didn't give us much to go on.

    Have you tried it and it was too slow? If so, what did you do? Where is the data coming from? What version of Postgres are you importing in to? Have you looked at partitioning the target table?

  3. #3
    Join Date
    Oct 2009
    Posts
    2
    Hi,

    Thanks for your reply. Here is the information.

    It importing 10,000 records per minute. I am just thinking on any performance tuning, may improve the insertion speed.
    FYI: All the data in text file. I am using java code to read the file and inserting it into the database directly. I am using postgres8.2 database version.

    I didn't know how to look at the partioning of Target table. Can you please help me on this too?

    Thanks in advance,

    Bala

  4. #4
    Join Date
    Sep 2009
    Posts
    39
    with ETL you can get about 20-80 M records/hour.

    I have data in home which table contains about 120 M records.

    for MS you have oledb driver

    Purchase

    or you can use pentaho etl.

    Pentaho Commercial Open Source Business Intelligence Kettle Project

  5. #5
    Join Date
    Aug 2009
    Location
    Olympia, WA
    Posts
    337
    If the data are already in a text file you should use Postgres' copy command.
    PostgreSQL: Documentation: Manuals: PostgreSQL 8.2: COPY

    On table partitioning
    PostgreSQL: Documentation: Manuals: PostgreSQL 8.2: Partitioning

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •