If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

 
Go Back  dBforums > Data Access, Manipulation & Batch Languages > Unix Shell Scripts > Pull only new files from Server A to Server B

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1 (permalink)  
Old 10-14-11, 11:41
aditya16in aditya16in is offline
Registered User
 
Join Date: Oct 2011
Posts: 2
Pull only new files from Server A to Server B

Hi,
I am very new to shell scripting and need to create shel script for following :

Look continuously for new files in server A(/hosting/apps) and as they arrive pull it to server B (/hosting/landing).

Note - only new files that have arrived in server A should be pulled into Server B. Script resides on Server B.
Also Server A is FTP only and RSYNC wont work.
Expecting to have 70 files dropped in a day on server A all starting with TTY_*

Thanks,
Adi
Reply With Quote
  #2 (permalink)  
Old 10-14-11, 12:03
kitaman kitaman is offline
Papabi's friend
 
Join Date: Sep 2009
Location: Ontario
Posts: 629
How frequently do you want to check for new files?
Assuming that both machines are unix/linux, the danger is that you will attempt to copy a file that is still in the process of being written, and only retrieve a portion of the file.
Is machine B allowed to rename/move/create files on machine A?
Do you have any control over the creation of files on machine A?
Is there a reason that you cannot push the files from machine A to B?
Reply With Quote
  #3 (permalink)  
Old 10-15-11, 01:09
aditya16in aditya16in is offline
Registered User
 
Join Date: Oct 2011
Posts: 2
files will be checked every minute.
We are not allowed to move/delete files from machine A as its not our property.

ALso we cannot have scripts residing on server A as we dont own it and are only allowed to pull files from there.

Can you think of any logic which can solve this issue. Any help will be appreciated.
Reply With Quote
  #4 (permalink)  
Old 10-15-11, 08:48
kitaman kitaman is offline
Papabi's friend
 
Join Date: Sep 2009
Location: Ontario
Posts: 629
Code:
#!/bin/ksh
if [ -r ftp.pid ]
then
   echo previous invocation still running
   exit
fi
echo $$ >ftp.pid
ftp A <<EOF >A.list
user
password
ls
quit
EOF
./createnewlist.ksh <A.list >new.list
sleep 10
ftp A <new.list
rm ftp.pid
If you can use .netrc files to store the userid and password, the script to compare the list of files on A and the ones that you already have will be simpler, and you will not have to put the userid and password in the new.list file.
The sleep for 10 seconds should allow any files that are being created while you are listing them to complete by the time you go back to get them.
Your application process should append files that you have successfully processed to your done list, giving you an opportunity to re-retrieve any files that did not process the first time.

createnewlist.ksh should look like:
Code:
#!/bin/ksh
rm new.list
while read perms x own size month day time file
do
   grep $file done.list
   if [ $? != 0 ]
   then
     echo get $file >>new.list
   fi
done<A.list
echo quit >>new.list
You may have to add additional reads or logic depending on the actual contents of A.list
Reply With Quote
Reply

Tags
linux, shell, unix

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On