Results 1 to 6 of 6
  1. #1
    Join Date
    Nov 2014
    Posts
    5

    Unanswered: Removing Headers from multiple files while concatenating

    HI,
    I have a requirement to merge multiple files into one.All the files are of the same type with same number of columns and exact column names as Header.
    When i merge all these files using cat command,i see the big file is getting all the header rows as well from multiple files.I tried many different ways but couldn't figure it out the header rows from the files.I only need one header row and the data from all the multiple files.

    I have test1 ,test2,test3 ....testn with all the same header row .

    cat > $working\test*.csv >$working\bigfile.csv

    when i concatenated the bigfile.csv getting all the header records as well.Can anyone please guide me hoe to rid of the header rows from second file onwards,i only need one header row in the bigfile.

    Thanks,
    Murali

  2. #2
    Join Date
    Sep 2009
    Location
    Ontario
    Posts
    1,057
    Provided Answers: 1
    Code:
    #part2
    read line
    while read line
    do
       echo "$line"
    done
    Code:
    #part1
    mv test1.csv output/bigfile.csv
    for file in *.csv
       do ./part2 <$file >>output/bigfile.csv
    done

  3. #3
    Join Date
    Jun 2003
    Location
    West Palm Beach, FL
    Posts
    2,713

    Cool

    Or.....
    Code:
    >output/bigfile.csv
    x=0
    for file in *.csv
       tail +$x $file >>output/bigfile.csv
       x=1
    done
    The person who says it can't be done should not interrupt the person doing it. -- Chinese proverb

  4. #4
    Join Date
    Feb 2004
    Location
    In front of the computer
    Posts
    15,579
    Provided Answers: 54
    or...
    Code:
    awk "1 == NR or 1 != FNR" *.csv >bigfile.csv
    -PatP
    In theory, theory and practice are identical. In practice, theory and practice are unrelated.

  5. #5
    Join Date
    Jun 2003
    Location
    West Palm Beach, FL
    Posts
    2,713

    Cool

    Quote Originally Posted by Pat Phelan View Post
    or...
    Code:
    awk "1 == NR or 1 != FNR" *.csv >bigfile.csv
    -PatP
    Nice.......
    The person who says it can't be done should not interrupt the person doing it. -- Chinese proverb

  6. #6
    Join Date
    Feb 2004
    Location
    In front of the computer
    Posts
    15,579
    Provided Answers: 54
    Quote Originally Posted by LKBrwn_DBA View Post
    Nice.......
    While AWK is very special purpose and the only thing it does really well is file transformations, it does that so well that not much else can compete. When you learn to use AWK declaratively (like this example), it can really "rock the house" for speed, simplicity, and elegance.

    -PatP
    In theory, theory and practice are identical. In practice, theory and practice are unrelated.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •