I've experienced out of memory problems when code I've written has gotten stuck in loops inifinitely or when I accidentally pushed the contents of the same array onto itself multiple times.

A few issues I have with this code. You can't ultimately guarantee the size of any of these files, so its best to not read them all in at the same time, you can process the file chunk by chunk using the scalar <> or the read() or even sysopen,sysread if you want to get complicated. instead of:
open(FILE, "<file.txt");
my @thing = <FILE>;
close FILE;
for(@thing) {
    if(/this/) {
        push @other,$_;
    }
    elsif(/that/) {
        push @another,$_;
    }
    else {
        push @crap,$_;
    }
}

[download]

try:
open(FILE, "<file.txt");
while(local $_ = <FILE>) {
    if(/this/) {
        push @other,$_;
    }
    elsif(/that/) {
        push @another,$_;
    }
    else {
        push @stuff,$_;
    }
}
close FILE;

[download]

Another thing that bothers me about this code is that you're not cleaning everything up and you have multiple instances of those files in memory at any given time. This could be adding up.. in the @filenames lets say it takes up 128kb, then you join it together on $filename without destoroying the array, you've used 256kb. if you want to operate on the scalar, then you might want to undef the array @filenames=(); to save some memory. Also, if you're just gonna join things anyways, why not use the str concatenation operator like so:
open(FILE,"<file.txt");
my $template = '';
while(my $line = <FILE>) {
    $template .= $line;
}
close FILE;

[download]

which I believe can be shortened even more into:
my $template = '';
open(FILE, "<file.txt") && while($template .= <FILE>) {};
close FILE;
die "bad stuff happened" unless length $template;

[download]

anyways, play with it somemore. and use offline mode to do some debugging and but 'print' statements all over your loops, Your data set may contain something you haven't thought of and it sounds to me like the program is looping infinitely because of it.

anyways, good luck with it, and let us know how it turns out.


-brad..

In reply to Re: OUT OF MEMORY! by reyjrar
in thread OUT OF MEMORY! by maddfisherman

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.