If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

 
Go Back  dBforums > Data Access, Manipulation & Batch Languages > Unix Shell Scripts > multi-thread a code

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1 (permalink)  
Old 09-12-08, 09:20
saccskiz saccskiz is offline
Registered User
 
Join Date: Feb 2004
Posts: 143
multi-thread a code

I am on a Sun Solaris 5.8
A piece of my load.ksh script has the following lines :

cat /data/dev/CODE_LIST1.TXT | while read column1 column2
do
genscript.ksh ${dbname} ${schemaname} member $column1 /data/dev/member_scripts/$column2
done

The CODE_LIST1.TXT looks something like

38222 38222_123456789.sql
48338 48338_438434845.sql


I am reading a CODE_LIST1.TXT file and passing the contents as variables into the genscript.ksh script
The problem is that CODE_LIST1.TXT could have 100 lines out there and currently our process if there are
100 entries in the .TXT file takes about 6 unavoidable hours just due to the processing involved.

Can I get some pointers on how I can multi-thread this code so that if there are more than 4 entries
in the CODE_LIST1.TXT, I want to split it up into 4 execution threads - each picking one up 1 line.
If there are 100 lines, each of the parallel threads need to pick up 25 each. If there are 105 lines,
then, first 3 picks up 27 each and the last one picks up 24 lines to process.
Can this be done ?
Thanks
Reply With Quote
  #2 (permalink)  
Old 09-15-08, 15:58
saccskiz saccskiz is offline
Registered User
 
Join Date: Feb 2004
Posts: 143
Got it

I figured this out...

linecount=`cat CODE_LIST1.TXT |wc -l`
echo $linecount
splitcount=`echo "scale=0; $linecount/4+1" |bc -l`
echo $splitcount
split -l $splitcount CODE_LIST1.TXT

## By the above, I now have 4 files - xaa,xab,xac and xad

for f in x*
do
parallelproc.ksh $dbname $schemaname $f &
done
wait


And in my parallelproc.ksh I have :

#!/bin/ksh
dbname=$1
schemaname=$2
filename=$3
cat /data/dev/$filename | while read column1 column2
do
genscript.ksh ....(with other parameters as in original posted thread)
done
exit
Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On