I am running perl code, computing some statistics based on lines of text. Each line of text is rather small, and as I watch the program progress with top (fedora) it never uses more than 4% of system memory. Without fail, after maybe 5 minutes, the program suddenly ends with a Out of memory! error. I have completely modified the input data in hopes of doing most of the processing beforehand, and have re-written the code several times in hopes of making it lighter. The code is now totally ugly and the error persists.

#!/usr/bin/perl

while(<>)
{
        chomp;
        @pages = split(/\+\+/, $_);
        @SonarData = ();
        %DOWN = ();
        @inlevels = ();
        @outlevels = ();
        @pagelevels = ();
        %OUT = ();
        %URL = ();
        %NUMOUT =  ();
        %LEVEL = ();
        $numpages = @pages;
        $intop = 0;
        $intotal = 0;
        $outtotal = 0;
        $inavg = 0;
        $outavg = 0;

        foreach $x (@pages)
        {
                ($id, $url, $level, $numout, $numin, $out, $in) = spli
+t(/\*\*/,$x);
                $pagelevels[$level]++;
                $inlevels[$level] += $numin;
                $outlevels[$level] += $numout;
                $OUT{$id} = $out;
                $URL{$id} = $url;
                $DOWN{$id} = 0;
                $NUMOUT{$id} = $numout;
                $LEVEL{$id} = $level;

                $intotal += $numin;
                $outtotal += $numout;
                $inavg += $numin*$level;
                $outavg += $numout*$level;

        }
        @pages = ();
        if($intotal)
        {
                $SonarData[6] = $inlevels[1]/$intotal;
        }
        else
        {
                $SonarData[6] = 0;
        }

        $total = 0;
        $numlevels = @pagelevels;
        $max = 0;
        $x = 0;
        while($x < $numlevels)
        {
                $total += $pagelevels[$x]*$x;
                if($pagelevels[$x]>$max)
                {
                        $max = $pagelevels[$x];
                }
        $x++;
        }

        $SonarData[0] = $total/$numpages;
        $SonarData[1] = $max/$numpages;

        if($pagelevels[1])
        {
                $SonarData[2] = $pagelevels[2]/$pagelevels[1];
        }
        else
        {
                $SonarData[2] = 0;
        }

        $SonarData[3] = $intotal/$numpages;
        $SonarData[4] = $outtotal/$numpages;

        if($intotal)
        {
                $SonarData[5] = $outtotal/$intotal;
                $SonarData[8] = $inavg/$intotal;
        }
        else
        {
                $SonarData[5] = 0;
                $SonarData[8] = 0;
        }
        if($outtotal)
        {
                $SonarData[9] = $outavg/$outtotal;
        }
        else
        {
                $SonarData[9] = 0;
        }
        $SonarData[10] = $SonarData[9] - $SonarData[8];

        if($intotal)
        {
                $max = 0;
                foreach $x (@inlevels)
                {
                        if($x > $max)
                        {
                                $max = $x;
                        }
                }
                $SonarData[11] = $max/$intotal;
        }
        else
        {
                $SonarData[11] = 0;
        }

        if($outtotal)
        {
                $max = 0;
                foreach $x (@outlevels)
                {
                        if($x > $max)
                        {
                                $max = $x;
                        }
                }
                $SonarData[12] = $max/$outtotal;
        }
        else
        {
                $SonarData[12] = 0;
        }

        @uplevel = ();
        @downlevel = ();
        @crosslevel = ();
        @sidelevel = ();
        $totalcross = 0;

        foreach $start (keys %OUT)
        {
                $startLevel = $LEVEL{$start};
                $startURL = $URL{$start};
                @links = split(/ /, $OUT{$start});
                foreach $end (@links)
                {
                        if($start != $end)
                        {
                                $endURL = $URL{$end};
                                $endLevel = $LEVEL{$end};
                                if($startLevel == $endLevel)
                                {
                                        $sidelevel[$startLevel]++;
                                }
                                elsif($startLevel < $endLevel && $star
+tURL =~ m{\Q^$endURL})
                                {
                                        print STDERR "downlink\n";
                                        $downlevel[$endLevel]++;
                                        $DOWN{$start} = 1;
                                }
                                elsif($startLevel > $endLevel && $endU
+RL =~ m{\Q^$startURL})
                                {
                                        print STDERR "uplink\n";
                                        $uplevel[$endLevel]++;
                                }
                                else
                                {
                                #       print STDERR "crosslink\n";
                                        $crosslevel[$endlevel++];
                                        $totalcross++;
                                }
                        }
                }
        }
        @tmp = keys (%DOWN);
        $total = 0;
        $count = 0;
        foreach $x (@tmp)
        {
                if(!$DOWN{$x})
                {
                        #a leaf page
                        $count++;
                        $total += $NUMOUT{$x};
                }
        }

        if($count)
        {
                $SonarData[7] = $total/$count;
        }
        else
        {
                $SonarData[7] = 0;
        }

        $SonarData[13] = $totalcross/$numpages;

        $N = 0;
        $S = 0;
        for($x = 2; $x <= 4; $x++)
        {
                $N += ($uplevel[$x] + $sidelevel[$x] + $downlevel[$x])
+;
                $S += $crosslevel[$x];
        }

        if($S)
        {
                $SonarData[14] = $N/$S;
        }
        else
        {
                $SonarData[14] = 0;
        }

        $SonarData[15] = ($uplevel[1]+$sidelevel[1]+$downlevel[1]+$cro
+sslevel[1])/$numpages;    $


        print "@SonarData\n";
}
[download]

sorry, i know the code is hideous, i've re-written it from scratch at least 4 times, its really wearing on my nerves. are there any common problems that lead to this kind of error?

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.