If the filesystem is really that busy, you want to avoid doing any kind of file system call as much as possible. So it makes sense to move all the file system calls as much back in your loop as possible.

In your loop you do the following file system calls:

  1.         my $mtime = stat("$local_dir/$file")->mtime;

    [download]
    2. my $age =(time - $mtime); chomp($age); # ??? why comp?! next unless ($age > 2); 
  2.         next unless (-f "$local_dir/$file"); # File was moved in the m
+eantime already?!

    [download]

You throw the result of these two file system calls away if the filename does not end on .xml. So a first step would be to only do the file system calls if the file name ends with .xml:

    while (defined(my $file = readdir($DH)) {
        next unless ($file =~ m/\.xml$/);

[download]

You also skip all files that are younger than 2 seconds, most likely under the assumption that they haven't been written yet. You could remember these filenames and just put them in a queue and check them again in 2 seconds time, instead of waiting for a second readdir round to produce them again. This would save you another call to opendir+readdir (which takes ages, as you say).

You also check if the file still exists before moving it - is your program running in multiple instances? Otherwise, you could remove that check as well.

In reply to Re: Read, SFTP Put and Move Files from a "Busy" NFS FileSystem by Corion
in thread Read, SFTP Put and Move Files from a "Busy" NFS FileSystem by longjohnsilver

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.