in reply to Out of Memory 2.

I'm sorry the responses here have come across as hostile, but to be honest, your presentation of the problem has been, um, problematic. (I'm truly grateful that someone -- you? -- fixed your original Out of memory. post; that entire thread is much easier to read now.)

I made some slight changes your program (as currently posted in the present thread), and it ran on a slightly modified version of the data provided in the earlier thread, with no "out of memory" problem.

I don't think my code changes had anything to do with memory management -- I just fixed the syntax errors, and changed to using STDERR for the status messages. In order to get it to work on the previously posted "postfix.txt" file (about 64KB), I had to delete the "." characters at the end of the data, in order to get past the "Bad $op ..." die condition.

#!/usr/bin/perl -w use strict; use warnings; $|=1; sub multiply { my( $ref1, $ref2 ) = @_; my @temp; for my $r1 ( @{ $ref1 } ) { for my $r2 ( @{ $ref2 } ) { push @temp, $r2 . $r1; } } return \@temp; } $/ = ','; my @stack; open POSTFIX, '<', 'postfix.txt' or die $!; while( my $op = <POSTFIX> ) { chomp $op; $op =~ tr/\r\n//d; ## remove any trailing newline if( $op eq 'x' ) { print STDERR "MULTIPLY\n"; push @stack, multiply( pop( @stack ), pop( @stack ) ); } elsif( $op eq '+' ) { print STDERR "ADD\n"; push @stack, [ @{ pop @stack }, @{ pop @stack } ]; } elsif( $op =~ m[^\d+$] ) { push @stack, [ pack 'v', $op ]; } else { die "Bad '$op' at position;" . tell( POSTFIX ); } } print STDERR "stacksize: " . @stack. "\n"; for my $item ( @stack ) { printf "%s", join ',', unpack 'v*', @{ $item }[ 0 ]; for my $group ( @{ $item }[ 1 .. $#$item ] ) { printf "+%s", join ',', unpack 'v*', $group; } print "\n"; }

With my version, you can redirect STDOUT to a file, and just see the status messages on your terminal. With a bash-like shell, you can also redirect STDERR to a separate file, which is what I did.

When the run finished successfully, the STDERR log file had 5170 lines containing "MULTIPLY" and 3966 lines containing "ADD", which happens to match the number of "x" and "+" characters in the data file, respectively. There were 30 elements in the "@stack" array, and 30 lines of data in the STDOUT file, the first two lines being:

1 2,5,6+2,4,6+2,3,6
The third line was over 8 KB long, the fourth line was:
40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55
The fifth line was about 150 KB long, the 7th was 658 chars, the 28th was 228 chars, and all the rest had a single 4-digit number. The number on the 30th line was "9166", which is happens to be the number of digit strings in the postfix.txt input file.

I have no idea what this sort of output is supposed to mean. Perhaps I've missed something, but I think your statements about the intent of your program, and the nature of your approach, consist so far of the following:

I wrote a program to sybolically process multiplications on an equation in postfix notation. (taken from the first post)

The goal is to convert "if-then-else" statements in a 4th GL scripting language into equations. For example "if(a) {}elsif (b) {} else {}" would becomes (a+b+c).

Once accomplished all paths through the 4th GL script can be found by "multiplying" the equations through. For example, two simple if/else statements could be represented as (a+b)(c+d.) The independent paths throught the code is: ac+ad+bc+bd.

The code provided here does this equation transformation. The difficulty is that the equation is 64K in characters and will result in a couple million independent elements as a result.

Sorry -- not only do I fail to get a clear idea of the goal, but I really can't see any relation between that description and the output that I got from your script. For that matter, I'm clueless about what the output data provides, let alone what the input represents. Still, I hope the information I provided was helpful in some way.

Replies are listed 'Best First'.
Re^2: Out of Memory 2.
by Anonymous Monk on Oct 27, 2008 at 01:45 UTC
    Thanks for the response. It is hard to take "Are you being deliberately thick?" as not hostile.

    The problem is large so I provided only a subset of what I am trying to solve, which appears confusing to folks. What I posted is an attempt to translate an algebraic solution into an expanded solution.

    When that failed in the previous post, for this post I was attempting to see what folks did in the way of "paging" when programs got too big for memory. It appears this question is to abstract for this forum.

    I've completed the first step which was to manually create array of arrays so I can page part of the data out to a database. With this I am up to 16 million array elements - only 8 million more to go. ;~)
    $|=1; $/ = ','; my @letters; my @words; $words[0]=0; my $idxop; open POSTFIX, '<', 'postfix.txt' or die $!; while( my $op = <POSTFIX> ) { $idxop++; chomp $op; chop $op if $op =~ tr[\n][\n]; ## remove any trailing newline print "$idxop ($op) WORDS: $#words; LETTERS: $#letters\n"; if( $op eq 'x' ) { print "MULTIPY\n"; ## GRAB LAST TWO ELEMENTS FROM WORDS BY RANGES x1->x2-1;x2->x3 +-1 my $x3=pop(@words); my $x2=pop(@words); my $x1=pop(@words); my $elements=$x3-$x1; ## APPEND NEW VALUES ON TOP OF STACK $idx=$x3; for (my $i=$x2; $i<$x3; $i++) { for (my $j=$x1; $j<$x2; $j++) { $letters[$idx++]= $letters[$j] . $letters[$i]; } print "AT: $i toward $x3 with $#letters\n"; } ## DELETE OLD ELEMENTS splice @letters,$x1,$x3-$x1; ## PUSH POINTERS TO ENDS OF NEW ALGOMATED ELEMENT push(@words,$x1); $elements=$idx-$elements; push(@words,$elements); } elsif( $op eq '+' ) { print " ADD\n"; ## GRAB LAST TWO ELEMENTS FROM WORDS BY RANGES x1->x2-1;x2->x3 +-1 my $x3=pop(@words); my $x2=pop(@words); my $x1=pop(@words); ## PUSH POINTERS TO ENDS OF NEW ALGOMATED ELEMENT push(@words,$x1); push(@words,$x3); } elsif( $op =~ m/^\d+$/ ) { print " STORE NUMBER $op\n"; push @letters, pack 'v', $op ; #push @letters, $op ; $words[$#words+1]=$#letters+1; } else { die "Bad '$op' at position;" . tell( POSTFIX ); } }