recording a stream of data remotely, and reliably

qbxk has asked for the wisdom of the Perl Monks concerning the following question:

hey all, i'm fishing for answers here, how would you tackle this problem, armed with perl of course.

I have a continuous stream of data coming into a machine, let's just say it's /var/filestream, none of this data can be lost however the machine that has the hardware to collect it has little diskspace - we have another machine on the same network with a huge disk we want to store this stream of data there.

What is the Right Way to record this? some stipulations:

no hardware changes. the big drive has to stay in the other machine and the little drive has to stay in the source machine.

no using the little disk as buffer space, i want as few disk writes to that disk as possible (why? it's not stable, it's a crappy disk, i don't want to use it repeatedly, because...)

the system must run flawlessly and without interruptions in the recording for...well, ever. months anyway...

the recorded stream will not be one huge file, it will be sectioned off into smaller files (with descriptive names) approximatley every 2-3 hours - again, no interruptions in the recording while the filename is being changed or the file buffer being written is truncated.

oh yeah, we're talking roughly a gigabyte and change per day, and it's always coming at a constant rate.

now, you might ask yourself what in sam's heck i'm doing here, but i don't want to tell you because it's a crazy setup we have due to budget constraints (we're a non-profit) we have to use the hardware on hand and this is what we've got. a machine that can sample, but can't store, and a machine that can store but can't sample

with that said, maybe i'll answer questions if you ask. thanks for helping out...

It's not what you look like, when you're doin' what you’re doin'.
It's what you’re doin' when you’re doin' what you look like you’re doin'!
- Charles Wright & the Watts 103rd Street Rhythm Band

Comment on recording a stream of data remotely, and reliably

Replies are listed 'Best First'.
Re: recording a stream of data remotely, and reliably by BrowserUk (Patriarch) on Jan 24, 2006 at 22:34 UTC
none of this data can be lost however the machine that has the hardware to collect it And if that machine dies? Within the constraint that that machine stays running, a RamDisk is as safe as any other part of memory in which to store the data. So, have the sampler process write the small files to a ram disk, and rename them to a known pattern when the file is complete. Have a second process monitoring the ramdisk for the known pattern, and ftp them to the remote machine as they appear. It then verifies (crc/md5) the transfer and deletes them from the ramdisk. Both the sampler and ftp processes would immediately fork on startup and the child would do the actual work. The parent simply monitors the child and should it abort for any reason, the parent immediately forks another copy and monitors that. At ~42MB/hour, a 128MB ramdisk would give you 3 hour window in which to rectify any problems with the remote machine or the intervening network. 256MB even better if the machine has the memory available. As a "last gasp" attempt to salvage a long term experiment, you could also arrange that if the communications with the remote machine are lost and the ramdisk is in imminent danger of being filled, then you could offload the ramdisk to the local harddrive just until the connection is restored. The weak link will always be the one machine with a dodgy disk though. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re: recording a stream of data remotely, and reliably by Corion (Patriarch) on Jan 24, 2006 at 21:16 UTC
I would try to broadcast the data over the network. By broadcasting the data, you leave the burden of completeness with the large machine and can bring in a second client in case you need to change the client program due to a bug you found, or because the HD ran full. The broadcast is easily implemented in UDP, but you could also use a plain TCP connection and sniff the traffic via Net::Pcap on the big machine to allow additional clients to tune in.	[reply]
Re: recording a stream of data remotely, and reliably by GrandFather (Saint) on Jan 24, 2006 at 21:22 UTC
Is there some current solution that you are trying to improve on or does the collection software need to be written from scratch? Can you not just share the large hard drive and write to it direct from the small drive machine? DWIM is Perl's answer to Gödel	[reply]
Re^2: recording a stream of data remotely, and reliably by qbxk (Friar) on Jan 24, 2006 at 21:47 UTC
No there's no solution right now, it's simply catting a file to disk, so a simple solution from scratch must be written. It's just convoluted since the disk happens to live in another machine... sharing is less simple in my opinion, but if it works it works, what technology would you pursue to accomplish that? the big disk is in a box running OS X 10.3 and the sampling machine is running Ubuntu 5.10, very bare install. I sense that NFS could do this, but i've never worked with it so i'm hesitant to do so. It's not what you look like, when you're doin' what you’re doin'. It's what you’re doin' when you’re doin' what you look like you’re doin'! - Charles Wright & the Watts 103rd Street Rhythm Band	[reply]
Re^3: recording a stream of data remotely, and reliably by GrandFather (Saint) on Jan 24, 2006 at 21:54 UTC
I don't know nix. In the windows world it would be a trivial matter of setting up a share on an appropriate folder from the large machine and accessing it from the small machine. I'm sure nix savy monks will be able to tell you how to do the equivelent in your context. If you can do it at all, then the minimum you need to do is change your cat to use the remote location. DWIM is Perl's answer to Gödel	[reply]
Re: recording a stream of data remotely, and reliably by cyclist38 (Hermit) on Jan 24, 2006 at 22:34 UTC
Since all you're doing now is cat > file.txt, why not consider using File::Remote? All it does is allow you to write to a local or remote file.	[reply]
Re: recording a stream of data remotely, and reliably by Tanktalus (Canon) on Jan 25, 2006 at 02:18 UTC
You know, depending on the non-profit, I would be sorely tempted to spend $500 out of my own pocket just to purchase a stable machine that could serve the purpose. Hundreds of GB of diskspace are cheap, and even if that's not big enough (at 1+GB/day one can still fill a "big" 200GB in only 6 months, so you may have a bigger "huge" disk than that), you can still have that brand new, stable, non-crappy disk for temporary writes. That said, if the two machines are stable enough themselves, NFS seems perfect to me. NFS is generally stable enough for most purposes.	[reply]
Re: recording a stream of data remotely, and reliably by chrism01 (Friar) on Jan 24, 2006 at 22:40 UTC
The simplest soln has to be NFS share the big disk. Anything else is going to be less stable/reliable. Doesn't mean it won't be good enough, just not as good and def more complex. If individual files are small enough (ie fit into small machine RAM easily), I'd consider multi-thread, 1 to listen to incoming stream, then Thread::Queue to output thread that sends file to big box. This may seem a bit daunting if you haven't done threading, but it's not that bad. Do you have some time in between each incoming file? If so, you may be able to do same thing (ie mem buff) without threads. In theory, even without sufficient mem to swallow 1 file, you could still use the threading method. Cheers Chris	[reply]