Results 1 to 3 of 3
  1. #1
    Join Date
    Nov 2013
    Posts
    1

    Unanswered: Shell Script - large files

    Hi Experts,

    I have two very large files (size in GB's). The every record which exist in first file, it should check for a match in 2nd file.
    Here we are reading second file again and again for every record in first file. if match found then write the matched record to
    other file.

    any help/ideas is appreciated.

    regards,
    H

  2. #2
    Join Date
    Jun 2003
    Location
    West Palm Beach, FL
    Posts
    2,713

    Cool

    Quote Originally Posted by hemanth24 View Post
    Hi Experts,

    I have two very large files (size in GB's). The every record which exist in first file, it should check for a match in 2nd file.
    Here we are reading second file again and again for every record in first file. if match found then write the matched record to
    other file.

    any help/ideas is appreciated.

    regards,
    H
    1) sort both files in the same order.
    2) perform Sort-merge join

    PS: Sorting both files is cheaper than anything you can do to 'unsorted' files.
    The person who says it can't be done should not interrupt the person doing it. -- Chinese proverb

  3. #3
    Join Date
    Sep 2009
    Location
    Ontario
    Posts
    1,057
    Provided Answers: 1
    The only caveat for sort is that sort creates temporary files. Make sure that the file system that sort uses for work files has enough free space.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •