I need a command line oneliner that will remove all the text at the begining of a text file up to the first blank line and save the results.

In fact I need to do this to multiply files in a folder.

I have tried the following ( and more) none work.

NOTE I don't care if the result ends in a new suffix ie: $file.out or not

EXAMPLES OF MY BAD NEWBIE CRAP:

cd ./txt/ ; for file in *txt ; do sed '1,/^$/d' $file ; done

cd ./txt/ ; for file in *txt ; do sed '1,/^$/d' 's/w\ \./tmp.tex' ; cp ./tmp.tex ./$file ; done

cd ./txt/ ; for file in *txt ; do sed '1,/^$/d' ; cp ./$file ./$file.out ; done

cd ./txt/ ; for file in *txt ; do sed '/^$/,/^$/D' "$file" ; done

This should do it

awk '/^$/ && ! textFound {next}{textFound=1; print}' file > newFile

Try this :

Hello again Jean Pierre. I know that the command above is correct but I have no idea why!

If I wanted to retain the lines from the first non-blank, to the last non-blank, I would do something like...

sed '/./,/./!d'

However, this would mean that blank lines at the end would be stripped out.

I have found that if I wanted to match to the end of the file, I would use...

sed '/./,/\$/!d'

I can't understand why I've had to escape the $ and now your example has totally confused me. Could you help clear it up for me?

Thanks, Damian

• Some explanations
  
  sed '/./,$!d'
  
  /./ <= Non empty line, at least contains a char
  $ <= End of file
  /./,$ <= Select lines from first nom empty to end of file
  ! <= Negate selection
  /./,$! <= Select empty lines at top of file
  d <= Delete selected lines
  /./,$!d <= Delete empty lines at top of file
  
  
  In a regular expression (/ . . . /)
  $ = end of line
  \$ = character $
  
  In address range (outside of RE)
  $ = last line
  
  
  
• To remove empty lines at end of file :
  
  Code:

  sed -e :a -e '/^\n*$/N;/\n$/ba' $file > $file.tmp

• To remove empty lines at top and end of file
  
  Code:

  sed '/./,$!d' $file | sed -e :a -e '/^\n*$/N;/\n$/ba  > $file.tmp

• Links
  do-it-with-sed
  Sed FAQ

Of course! It's an address, not an RE. I still don't understand why I have to escape the $ in the following...

sed '/./,/\$/!d'

As you see, this is a regex and like you, I believe that this ought to represent the character $. It doesn't btw.

If your file dosen't contains the character '$' the address /\$/ is take as $ (end of file).
In that case /./,$ and /./,/\$/ are equivalent.

That seems to be correct. Is this behaviour documented anywhere? My man page has the following to say about addresses...

Certain commands called addressed commands allow you to specify one line or a
range of lines to which the command should be applied. The following rules apply
to addressed commands:

o A command line without an address selects every line.
o A command line with one address, expressed in context form, selects each
line that matches the address.
o A command line with two addresses separated by commas selects the entire
range from the first line that matches the first address through the next
line that matches the second. (If the second address is a number less than
or equal to the line number first selected, only one line is selected.)
Thereafter, the process is repeated, looking again for the first address.

My man page says the same thing.
The document do-it_with-sed specify :
This last point can explain this not documented behavior

Extract from Sed FAQ

Thankyou.

Geek alert! Geek alert!



And that, gentlebeings, is why I personally don't use sed.

Oh, it's damm powerful, as you can plainly see, but it's incomprehensible. At least in my experience, when I come back to a sed-line that I myself have written, even one day later, I have forgotten what it means and I spend a long time puzzling it out.

Witness the fact that one line of 'chicken scratches' was followed by about four explanatory posts describing what it means. To me, that's a maintenance issue.

I'm not saying that awk is too much better at this than sed, but at least you have the opportunity to put some comments into it. And you can also write more than one rule, you can write procedures and so-forth, which make the whole process much easier to understand when you encounter your own code a second time.

'Chicken scratching' certainly tends to give Unix shell programming a bad reputation.

... P.S.: Nothing personal intended here! Nothing at all. Just another point of view.

hi Guru

I agree with you.
I use sed especially to carry out substitutions, for the rest I use awk.

Did this.....

cd ./txt/ ; for file in *txt ; do sed '/./,$!d' $file > $file.tmp ; mv $file.tmp $file ; done

Got this...

bash: syntax error near unexpected token `do'


I don't see anything wrong here... except maybe it needs quotes around the file names etc.. so...

I did this...


cd /home/Wolfe/Mail/IRCUNDERGROUND/txt/ ; for file in *txt ; do sed '/./,$!d' "$file" > "$file.tmp" ; mv $file.tmp $file ; done


No errors, but... the headers remain.

• Put quotes around all file names
  
  Code:

  cd /home/Wolfe/Mail/IRCUNDERGROUND/txt/
  for file in *txt
  do
     sed '/./,$!d' "$file" > "$file.tmp"
     mv "$file.tmp" "$file"
  done

• Execute 'set -x' before the for loop to verify commands
• If your directory doesn't contains file '*txt' the variable 'file' will get the value '*txt'.
  To avoid that, you can use 'find' command :
  
  Code:

  cd /home/Wolfe/Mail/IRCUNDERGROUND/txt/
  find . -name '*txt' | \
  while read file
  do
     sed '/./,$!d' "$file" > "$file.tmp"
     mv "$file.tmp" "$file"
  done

Well.. I did that and the verdict is...

No errors... and no results.
the files remain as they were as if I had done nothing...

Some other things that I have tried:

cd /home/Wolfe/Mail/IRCUNDERGROUND/txt/ ; for file in *txt ; do perl -e '$i=0; while(<>){if($i|^[A-Za-z]:|/^\b*$/){print $_};$i++}' < $file > $file.tmp ; done

All this does is crash.. symbols within the the text file tend to screw things up "<" for example will cause an error.. so will pipes "|" ++> etc...

My thinking was that any line in the text that begins with "<text>:" is most likely to be a header, and therefore the line could be deleted. This would be better then just deleting to the first blank line, because if someone forwards an email as text to the system, the headers from the first would show up in the post. Problem is however, not all the lines in the header start with a string and a ":", some start with a "<" etc.. and therefore screw things up and don't get deleted. I pre-deleted most of the symbols causing this, but it is a pointless task.

The things that is really ticking me off. is the I did have a working sed script oneliner that did the job. It deleted everything to the first blank line and echo'd the contents to a new file ($file.out) but somehow it got deleted even in my backups. Oh well back to the drawing board as they say.

Sorry for the long rant.