Good Day to all of you Enlightened Monks,

I've been trying to find the best solution to perform some simple yet complicate (performance-wise) tasks on a "busy" NFS file system.

Let me be clearer: I'm on a Linux Debian Box and the filesystem is mounted as follows:

/mnt/mounted_dir/ type nfs (rw,noatime,rsize=32768,wsize=32768,hard,intr,tcp,nfsvers=3,addr=x.x.x.x)

By "busy" i mean that at "random" times each day a lot of (maybe 15 x second) small .xml files (~2k each) get written on this filesystem.

The things i need to perform on these files are the following:

1. Read/Fetch each file as soon as it has been written completely.
2. SFTP-Put the file on a remote FTP server.
3. Make sure the transfer has been completed successfully
4. Move the local copy of the "transferred" file to another local directory.

These tasks don't look really complicated at first sight but, i have to say, that when the filesystem finds itself in this "busy" state even executing a simple "ls" takes ages, really ages. The script only works smoothly when the filesystem is not in a "busy" state.

Here's the core snippet of my code for anyone interested to helping me out in making things perform better. Thanks. 

#!/usr/bin/perl -w

use strict;
use diagnostics;
use autodie;
use Net::SFTP::Foreign;
use File::Copy;
use File::stat;

-->> omitted code to make things more readable 

$sftp = Net::SFTP::Foreign->new($host, %args);
$sftp->setcwd($remote_dir) || die log_msg($sftp->error."Exiting...\n")
+;

opendir($DH, $local_dir) or die $!;
    while (defined(my $file = readdir($DH)) {
        my $mtime = stat("$local_dir/$file")->mtime;
        my $age   =(time - $mtime);
        chomp($age);
        next unless ($age > 2);
        next unless (-f "$local_dir/$file");
        next unless ($file =~ m/\.xml$/);
        
        # sftp put section
        if ($sftp->put("$local_dir/$file")) {
          move("$local_dir/$file", "$local_dir_mv") or die log_msg("Th
+e move operation failed: $!");
        }  
        else {
           die log_msg($sftp->error);
        }
    }   
closedir($DH);

[download]

In reply to Read, SFTP Put and Move Files from a "Busy" NFS FileSystem by longjohnsilver

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.