SpaceCake has asked for the wisdom of the Perl Monks concerning the following question:

I have 2 small troubles, historically speaking my usage of perl is under 3 months, so please indulge my ignorance. I have a srv on which I constantly via ftp receive tar files. I will need them to:

- untar

- ftp to remote host for processing

- remove untared items

- move the processed tars to another directory("old_data/")

Problem which I have with the current script is that I don't know which files are complete and which ones are just being uploaded, and I after the successful untar, move them further. Ideas ?

use Archive::Tar;

use File::Copy;

chomp(my @list=`ls data*.tar`);

foreach $file (@list) {

$tar = Archive::Tar->new;

$tar->read( $file );

$tar->extract();

UPDATE

Any chance to help me out with this one, its an another idea how aproch tar, but in my case it does not work, help ? :(




#! /usr/bin/perl -w
use Net::FTP;
use File::Copy;
my $sfile = "data*.tar"
my $dstftp = "1.1.1.1"
chomp(my @files=`ls $sfile`);
foreach $files (@files) {
        copy("$files","work/$files") or die "Copy failed: $!";
        my $testit = exec "cd work/ ;tar xf $files ; cd ..";
        if ($testit == undef)
                {
                chomp(my @lpfiles = `ls work/data_10*`);
                foreach $lpfiles(@lpfiles) {
                        $ftp = Net::FTP->new("$dstftp", Debug => 1)
                                or die "Cannot connect to host: $@\n";
                        $ftp->login("stt",'stt111')
                                or die "Cannot login \n ", $ftp->message;
                        $ftp->binary or die ("cant Binary \n");
                        $ftp->hash or die ("cant Hash \n");
                        $ftp->cwd("test/")
                                or die "Cannot change working directory \n", $ftp->message;
                        $ftp->put("work/$lpfiles")
                                or die "put failed \n", $ftp->message;
                        $ftp->quit;
                        move("work/$lpfiles","/dev/null");                     
                        }
                        move("$files","/dev/null");
               }
                else {
                	print "Nothing to do. No readable TAR files"
        }
}

Replies are listed 'Best First'.
Re: How to handle broken tars ?
by davido (Cardinal) on May 21, 2004 at 15:55 UTC
    Use two passes. In the first pass over the directory, make a list of all files in the directory and their size. Ten minutes later, if that size hasn't changed, move them to wherever it is you want.

    The logic, in more detail, might look like this:

    • Read the directory. For all new items, add them to your in-process log.
    • For all pre-existing entries, determine if their size changed since they were logged ten minutes ago.
    • If size didn't change, they're ok to process. Do whatever processing you need to do, move them to wherever they need to go, and remove their entry from the log (or mark it processed).
    • If size did change, write new size to the log so that file can be revisited when the script runs again in ten minutes.

    This doesn't guarantee the tar's aren't corrupt, but it does at least give them time to finish being sent.

    Update:

    Another strategy might be to attempt to get a flock on the tar. I'm suggesting this under the assumption that the FTP server locks the file while it's still being written to. If that's the case, you won't be able to get a flock on the file while it's being uploaded. And that will tell you whether or not it's safe to process it.


    Dave

      ftpwho

      give's me the all the ftp account(s) which are logged in .. also STORE calles where the give partially the filename ..

      unfortunatly the file name aint complete, they are truncated, thatts bad, because all the incoming files have timestemps on them, so I can't even quess them :(

Re: How to handle broken tars ?
by Stevie-O (Friar) on May 21, 2004 at 15:13 UTC
    I think the best method is to have people upload to one directory, and when the upload completes, move the tar to another directory. I believe PAUSE/CPAN does something like this, so maybe you can find out how they do it
    --Stevie-O
    $"=$,,$_=q>|\p4<6 8p<M/_|<('=> .q>.<4-KI<l|2$<6%s!<qn#F<>;$, .=pack'N*',"@{[unpack'C*',$_] }"for split/</;$_=$,,y[A-Z a-z] {}cd;print lc
      nope, can't do it like that. client software is automatic and not reachable to my person. :(
Re: How to handle broken tars ?
by tbone1 (Monsignor) on May 21, 2004 at 15:47 UTC
    Hmm. You might have to watch a file's size and, when it doesn't change between checks of, say, five minutes, assume it's done. That's kinda ugly, but it's the best I can think of before my meeting.

    --
    tbone1, YAPS (Yet Another Perl Schlub)
    And remember, if he succeeds, so what.
    - Chick McGee

      Can I somehow check if it give's me the output at all to tar extract ? and use that one like a check ?