none of this data can be lost however the machine that has the hardware to collect it

And if that machine dies?

Within the constraint that that machine stays running, a RamDisk is as safe as any other part of memory in which to store the data.

So, have the sampler process write the small files to a ram disk, and rename them to a known pattern when the file is complete.

Have a second process monitoring the ramdisk for the known pattern, and ftp them to the remote machine as they appear. It then verifies (crc/md5) the transfer and deletes them from the ramdisk.

Both the sampler and ftp processes would immediately fork on startup and the child would do the actual work. The parent simply monitors the child and should it abort for any reason, the parent immediately forks another copy and monitors that.

At ~42MB/hour, a 128MB ramdisk would give you 3 hour window in which to rectify any problems with the remote machine or the intervening network. 256MB even better if the machine has the memory available.

As a "last gasp" attempt to salvage a long term experiment, you could also arrange that if the communications with the remote machine are lost and the ramdisk is in imminent danger of being filled, then you could offload the ramdisk to the local harddrive just until the connection is restored.

The weak link will always be the one machine with a dodgy disk though.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re: recording a stream of data remotely, and reliably by BrowserUk
in thread recording a stream of data remotely, and reliably by qbxk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.