Monks,

I have a script that pulls a large queue of files from one of my servers to another. These files range in size from a few hundred KB to up to a bit over 50 GB.

As my title suggests, I'm doing this copying using the File::Copy module's copy sub out to a network path (NFS). On a couple occasions this has failed on the larger files (30GB+). By failed I mean copy returned a 0. Unfortunately this was a quick and dirty script with minimal logging so I didn't get the actual error back from the copy sub. I have since added better logging, but have not reproduced the problem.

The script works using Parallel::ForkManager to fork out a bunch of processes to handle the individual downloads. Of course these larger files would be transferring longer than the others meaning they're on the network longer and more vulnerable to network disconnects/issues. I have checked for network connectivity issues in the logs and such with no luck, but it would've definitely been obvious to one of my admins if either of these servers was having problems with a network connection since they feed some maximum up-time apps.

This is on Windows systems (ActiveState), so I took a quick peek at File::Copy's source and it looks like it just links out to the OS specific file copy routines. So no real issue there. I'm using the default copy settings: not changing the buffer size, etc.

Anyway, I just thought I'd post and get your collective monkly opinion before shelving this and waiting for the logs to catch something.

In reply to File::Copy on Large-ish Files over Network by temporal

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.