Instead of scheduling via cron at predefined intervals, take a different route. A perl script running in the background as service, polling the directory for accumulated files for instance every 15 seconds. If the number of files is lower than a given threshold, say 1000 files, and the time elapsed since last proccessing of files (moving around)is less than two minutes, do nothing. If the threshold is reached and time elapsed is less than 2 minutes, perform the operation regardless. Else, if the time threshold goes over two minutes, and the files are less than the threshold, perform the operation.

The operation most likelly will consist of spawning a child proccess to do the task. If the files aren't actually moved from the watched folder, an internal list kept by the parent proccess could be kept, passing down to its childs related information.

The system could also be deviced as fail safe. Having a different Perl proccess monitoring the working parent. If the parent fails, automatic restart were it left off and alerting of the admnistrative team.

In reply to Re: File::Copy and file manipulation performance by psychotic
in thread File::Copy and file manipulation performance by Fendaria

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.