in reply to recording a stream of data remotely, and reliably

none of this data can be lost however the machine that has the hardware to collect it

And if that machine dies?

Within the constraint that that machine stays running, a RamDisk is as safe as any other part of memory in which to store the data.

So, have the sampler process write the small files to a ram disk, and rename them to a known pattern when the file is complete.

Have a second process monitoring the ramdisk for the known pattern, and ftp them to the remote machine as they appear. It then verifies (crc/md5) the transfer and deletes them from the ramdisk.

Both the sampler and ftp processes would immediately fork on startup and the child would do the actual work. The parent simply monitors the child and should it abort for any reason, the parent immediately forks another copy and monitors that.

At ~42MB/hour, a 128MB ramdisk would give you 3 hour window in which to rectify any problems with the remote machine or the intervening network. 256MB even better if the machine has the memory available.

As a "last gasp" attempt to salvage a long term experiment, you could also arrange that if the communications with the remote machine are lost and the ramdisk is in imminent danger of being filled, then you could offload the ramdisk to the local harddrive just until the connection is restored.

The weak link will always be the one machine with a dodgy disk though.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re: recording a stream of data remotely, and reliably