in reply to Out of Memory 2.

As Illuminatus points out, there are better ways to post your question (or perhaps, better questions you should be asking).

"Out of memory" failures should be pretty rare, in general, and only show up when there's some bug in the code, and/or some misunderstanding about the nature and quantity of your input data, so a few words to describe the data and what the code is supposed to do with it would be relevant.

A common cause of "Out of memory" failures is the use of a variable as an array index, when the value of the variable happens to be something unexpectedly (unintentionally) large -- here's the quickest/easiest way to get "Out of memory":

$ perl -le '$x=2**30; print $x; $a[$x]=0' 1073741824 Out of memory during array extend at -e line 1.
(Though conceivably some perl installs on some machines might succeed with that particular example.)

Another common cause is making multiple (unnecessary) copies of large files in memory -- e.g. doing stuff like:

@lines = <FILE>; # copy 1 $all = join("",@lines); # copy 2 for (@lines) { ($key,$val) = split " ", $_, 2; $hash{$key} = $val; # copy 3 }
If you know in advance that your script needs to access to more data than can fit in memory at a given time, then SQL access to a database is certainly a good way to work around that. DBM files are another option, if the data are structured as "key:value" tuples.

Whatever you are trying to accomplish, I would not recommend trying to game it so that your process tries to consume as much memory as possible without taking "too much" -- that approach only worked back in the old days of single-job systems (the original MS-DOS, DEC's old RT-11 are the only ones I recall), and nobody uses those anymore.

Assuming you know enough about your data and your task, you should be looking for a solution that runs within reasonable memory limits (or you should be putting more RAM on your machine until it's not a problem anymore).

Replies are listed 'Best First'.
Re^2: Out of Memory 2.
by dneedles (Sexton) on Oct 25, 2008 at 20:13 UTC
    Thanks for the info. I updated the initial post to provide a bit more tactical info and to consolodate the answers to the many questions.
      Could you expand a little on what you're trying to do? That is to say, Matrix manipulation, Fourier Transforms, 3-D animation, etc.?

      There may already be a CPAN module optimized to the problem you're trying to solve.

        The problem is taking a simple alegebraic equation involving only (), +, and * and expanding it by muliplying through to remove the ().

        For example: (a+b+c)*(d+e) becomes: ad+bd+cd+ae+be+ce.

        The issue is the equation is 64K and the result 3 million plus independent elements. So any package would need to handle some way to deal with the RAM blow up since it appears PERL's lack of ability to page heap is what is resulting in the "out of memory!" error.