Per line processing

mperham · #1 (**permalink**) 01-05-05, 12:32

I'm trying to write a script which scrubs illegal characters from an XML file. Each line is essentially a db record and I would like to check each line for illegal characters. If the record has illegal characters, I want to remove that line from the file and put it in another error report file.

I have written the script to do all this but it is quite slow, I'm assuming due to process creation overhead. Here is the working script:

Code:

while read DIRTY ; do
        CLEAN=`echo "$DIRTY" | tr -d '\001'-'\010''\013''\014''\016'-'\037''\177'-'\377'`
        if [ "$CLEAN" != "$DIRTY" ]; then
                echo Found a line of bad XML
                echo "$DIRTY" >> $1.ERR
        else
                echo "$DIRTY" >> $1.NEW
        fi
done < $1

It currently takes 3 minutes of CPU time to process a 25MB file with 120000 lines. Perhaps two calls to sed (one to search for lines with bad chars, the other for lines without) would work but I don't know the actual sed commands to do that. Any help?

mike

mperham · #2 (**permalink**) 01-05-05, 13:22

I've been trying to get sed working instead and have a pretty good start:

Quote:

# Find bad lines
sed -n -e 's/(^*[\\001-\\010\\013\\014\\016-\\037]*$)/ \\1/p' $INPUT > $INPUT.ERR
# Find good lines
sed -n -e 's/(^*[^\\001-\\010\\013\\014\\016-\\037]*$)/ \\1/p' $INPUT > $INPUT.NEW

But I get this error:

sed: -e expression #1, char 48: Invalid range end

Anyone know what the problem is?

vgersh99 · #3 (**permalink**) 01-05-05, 13:55

how about a single 'awk':

nawk -f mp.awk input.txt

here's mp.awk

Code:

BEGIN {
  #PAT_bad="([a-c])|([xyz])"
  PAT_bad="([\\001-\\010])|([\\013\\014)|([\\016-\\037])"
}
{
  out= FILENAME "." (match($0,PAT_bad) ? "bad" : "good");
  print > out;
}

mperham · #4 (**permalink**) 01-05-05, 14:34

Nice, vgersh99. I didn't know this but many of the GNU tools that deal with regexp have preset character sets that you can use in your regexps. Here's a solution that I came up with that seems to work also:

Quote:

# Strip any DOS linefeeds - they will screw up the search for control characters
sed 's/^M$//' $1 > $1.CLEAN
rm $1
mv $1.CLEAN $1

grep '[[:cntrl:]]' $1 > $1.ERR
grep -v '[[:cntrl:]]' $1 > $1.NEW

That takes 14 seconds instead of 3+ minutes. Thanks for the tip.