The mystery 300MB file is a trimmed, translated, and sorted EDI dataset. Although it would make much more sense to divide the file up by daily transactions, it is just a little to risky for me - as the file is appended on by several different other scripts. The file (which I lovingly refer to as "The Mother Log") is not only written to by several other Perl scripts, but it is also accessed by several cgi scripts (not all Perl), and queried by Access (lol). I would prefer to load the file into memory once a day, and then when asked for - it grabs the info out of Ram, instead of thrashing about the HD.

UPDATE: I've decided for now that I'll rotate the log more often in order to keep it below 150MB, and setup a RamDisk that will mirror the backup file. I think this setup is best for now for several reasons:

1. The log isn't meant to replace existing datasets, but it is meant to remove duplicate transactions in order to keep the file size down (one day's worth of EDI data for my company is around 700MB, the MotherLog will be about 150MB - and will contain a month's worth of transactions).
2. I don't want to rearrange the data too much because the log has in the past been used to extract charge backs from suppliers breaking their contracts. I'd rather the file remain extremely simple so that errors can be recognized immediately.
3. I'm not getting paid enough to reprogram this thing again :)

Thanks everyone!

In reply to The Mother Log by foxops
in thread Sharing a 300MB variable. by foxops

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.