I can't say enough about using end of file markers to ensure that the file is intact. The 10 minute wait works great when you're on a stable network, and the two machines are nearby, but when you start getting to larger files, and systems further apart, you run the risk of the transfer dying in the middle. This is especially bad if you can't back out of whatever you're about to do if you start processing.

I've done a number of different tricks over the years. For those times when you control the format (and you can't change to SOAP, or something that's intented for this sort of thing), I make sure that all files have the last line of END (or something else that will never show up in the file)

If you can't modify the format, and can't easily test if the file is complete, I'll send two files -- if the important file (the one you actually care about) is filename, the second file is file filename.done, with no real data in it.

In reply to Re^2: Combining Data Files by jhourcle
in thread Combining Data Files by royg

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.