hey all, i'm fishing for answers here, how would you tackle this problem, armed with perl of course.

I have a continuous stream of data coming into a machine, let's just say it's /var/filestream, none of this data can be lost however the machine that has the hardware to collect it has little diskspace - we have another machine on the same network with a huge disk we want to store this stream of data there.

What is the Right Way to record this? some stipulations:

  • no hardware changes. the big drive has to stay in the other machine and the little drive has to stay in the source machine.
  • no using the little disk as buffer space, i want as few disk writes to that disk as possible (why? it's not stable, it's a crappy disk, i don't want to use it repeatedly, because...)
  • the system must run flawlessly and without interruptions in the recording for...well, ever. months anyway...
  • the recorded stream will not be one huge file, it will be sectioned off into smaller files (with descriptive names) approximatley every 2-3 hours - again, no interruptions in the recording while the filename is being changed or the file buffer being written is truncated.
  • oh yeah, we're talking roughly a gigabyte and change per day, and it's always coming at a constant rate.

    now, you might ask yourself what in sam's heck i'm doing here, but i don't want to tell you because it's a crazy setup we have due to budget constraints (we're a non-profit) we have to use the hardware on hand and this is what we've got. a machine that can sample, but can't store, and a machine that can store but can't sample

    with that said, maybe i'll answer questions if you ask. thanks for helping out...


    It's not what you look like, when you're doin' what you’re doin'.
    It's what you’re doin' when you’re doin' what you look like you’re doin'!
         - Charles Wright & the Watts 103rd Street Rhythm Band

    In reply to recording a stream of data remotely, and reliably by qbxk

    Title:
    Use:  <p> text here (a paragraph) </p>
    and:  <code> code here </code>
    to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.