Help with replacing string in a file

dusoo · #1 (**permalink**) 05-23-08, 03:41

Hi everyone.
i would like to read file - line by line,
search for a string (with regular exp)
if found, replace it with something retrieved by a function.
Thanks for any ideas...

Pat Phelan · #2 (**permalink**) 05-23-08, 14:44

Perl, AWK, PHP ? The part about the function lets SED off the hook, but there are still many contenders!

-PatP

dusoo · #3 (**permalink**) 05-26-08, 10:39

Hi,... You were not really helpfull as i expected...
Could u write an example how to replace one string in a file if matches a pattern?
thanks
(shell)

stolze · #4 (**permalink**) 05-27-08, 00:08

Have a look at Pat's suggestions and try to understand the basics of the different tools and programming languages. It is really straight-forward with Perl, for example - with the others as well, - and you can find tons of examples if you put your favorite search engine to work. Why don't give it a try?

dusoo · #5 (**permalink**) 05-27-08, 02:36

Hi, of course i have tried pat's suggestions...

below is the code i came up with. It's working, but it takes a lot (an hour..) to process one single (50megs "large") file. I have later even larger files to be processed...
Is there a way to faster it a bit?
Regarding the input file. There is maybe each fifth line containing the "mailD,id" string, that i will replace by other value.

thanks for any help.

Code:

#!/bin/bash

FILE=$1
while read line
do
  if [ `echo "$line" | grep "mailD,id" | wc -l` -eq 1 ]
  then
		tmpmail=`echo $line | perl -p -e 's/^.*mailD,id = ([A-Z0-9][A-Z0-9]( [A-Z0-9][A-Z0-9])+),.*$/\1/'`
  	lda=`ldapsearch -a always -D cn=$user -w $pass -h 10.16.41.10 -p 16611 -s sub -L -b "id=$tmpmail,domainName=subsD,O=Test,C=MM" objectclass=common 2> /dev/null | grep mailo | perl -p -e 's/mailo: ([0-9]+)/\1/'`
   	perl -pi -e "s/$tmpmail/$lda/" log.log
  fi 	
done < $FILE

mike_bike_kite · #6 (**permalink**) 05-29-08, 10:36

You could help us a little bit by putting a few comments in your code that explains what each part does. It takes a lot of time to try and decipher other peoples code and remember we're not being paid to do it. These are my guess's on how to make your code faster.

You're currently processing each line of your large file but really you just want to process those lines containing "mailD,id". then to get a big improvement in speed you want to try and do things via pipes rather than via loops. Why not use sed to extract this info outside the loop and just do it once for each user ie

Code:

#!/bin/bash

FILE=$1

grep "mailD,id" $FILE | \
    sed 's/^.*mailD,id = ([A-Z0-9][A-Z0-9]( [A-Z0-9][A-Z0-9])+),.*$/\1/'` | \
    sort | \
    uniq | \
    while read tmpmail
    do
  	lda=`ldapsearch -a always -D cn=$user -w $pass -h 10.16.41.10 -p 16611 -s sub -L -b "id=$tmpmail,domainName=subsD,O=Test,C=MM" objectclass=common 2> /dev/null | grep mailo | perl -p -e 's/mailo: ([0-9]+)/\1/'`
   	perl -pi -e "s/$tmpmail/$lda/" log.log
    done

exit

Before posting back that my code doesn't work etc please remember it's difficult to read your original code, there is no test data and no explanation of what's supposed to happen. You may need to use \( instead of ( in the sed command. This should be faster.

Mike