Kozz...

The part of your post that sends up a red flag for me is that you say your app gets incredibly slow, though not CPU intensive. If you're talking about really large arrays, it's likely that you're running out of real memory and you're getting a lot of disk thrashing due to virtual memory.

One big memory waster I see in your method is that you're reading the whole file into memory at once. Since all your records are separated by blank lines, you can take advantage of Perl's "paragraph mode". Just set $/ to the null string, like so: 

$/ = '';

[download]

and every $var = <FILE> will read in one record.

Check perldoc perlvar for more information on $/.

If that still doesn't take care of it, you might consider using some kind of database for temporary storage. They usually do some kind of smart caching in an attempt to keep the stuff you need in memory and the stuff you're not using on disk. I'm not particularly knowledgeable on databases, so I'll defer that question to others.

Have fun!

*Woof*

In reply to Re: Efficiency and Large Arrays by splinky
in thread Efficiency and Large Arrays by Kozz

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.