Without seeing the code in question (or at least something stripped down that produces similar bloat under the different perls) I don't know if you're going to get a good response.

Having said that, one alternative when you start running out of RAM when processing a hash is to start tossing the data into something on disk using BerkeleyDB or the like. You'll lose some speed but you shouldn't hit the same memory wall.

Ah, good point. This msaaivley cut down version still produces the same bloat.

Rewriting the code is an option, but not one I'd like to go into unless there is no other hope.

#!/usr/bin/perl -w

use strict;

my %a = ();
my $res = `ps v $$`;
print "$res\n";

for my $line ( 1 .. 19000 ){
  for ( "AA" .. "DZ" ){
     $a{$line}{"$_$line"} = $line;
    }
 }

$res = `ps v $$`;
print "$res\n";

exit 0;
[download]

---

'I think the problem lies in the fact that your data doesn't fit my program'.

Maybe with your cut down script, Devel::Size could provide some insight (at least give you a diff on the two architectures between the structure size and the structure+data size).

-derby

The code is identical. We're trying to prove that everything works exactly as it did, before we migrate over.

Any thoughts on this being to do with 32bit v 64bit architecture?

---

'I think the problem lies in the fact that your data doesn't fit my program'.

64 bit is certainly going to use more memory. Your integers will be bigger. Also, you're using 5.8.2 on the new box and 5.6 on the old. That is likely to make a difference.

Another problem could be that the new machine is shipping with soft/hard ulimits set differently by default. Check ulimit on the command line before running. Tweaking the hard limits may require a kernel recompilation.

Good luck!

HTH, --traveler

You might want to compare your users memory quotas on those mashines (you might need a sysadmin for that).

Also, you might check what applications are running on those mashines. It just might be that there is another app that eats a lot of memory on the AIX 5.3.

HTH.

Also take a look at any ulimit settings, or anything else that may limit how much memory is available to a process. Can you see if the new process is actually using more memory or running into other limitations?

--
sgifford's Web page

I've hijacked a friendly sysadmin bloke - the ulimits are set the same way, there was no other process running on either box during these tests. I've just stumbled into a big problem though - I don't think I'm going to be able to recompile the Perl without major internal wranglings over support on the box. Arrrrrrgh.

---

'I think the problem lies in the fact that your data doesn't fit my program'.

As an irish joke goes: "could you tell me the way to tipperary?" - "well, I wouldn't start from here".

I don't have an idea either, but I see too many changes to identify "the guilty party": OS release change, hardware change, perl release change. To make shure it isn't (or is) perl's fault I would install the same version of perl (5.8.2) on the old box, run the program and start from there.

--shmem

_($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

The problem is that the old box is the current production machine. No-one would let me go anywhere near changing that. It's beginning to look like I either need new Perl binaries, which they just might let me load, or I re-write my routine.

---

'I think the problem lies in the fact that your data doesn't fit my program'.

Then do it the other way round. Build a 5.6.0 perl on the new box, and test that.

--shmem

_($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

Compile perl with a prefix, and store it in your home directory, where nobody else can touch it. It should be safe.
-nuffin
zz zZ Z Z #!perl

My program was sucking up double as much memory and this only because of differences in Perl versions.
Same compilation flags, same way of installing.

I've come to that conclusion myself. The keys to my main hash are stupidly big, so as a quick fix I'm writing them to disk as individual files, containing just a unique number. I'm then using that number as the key. It has reduced the memory usage, but not by as much as I'd hoped.

Thanks one and all for your help on this.

---

'I think the problem lies in the fact that your data doesn't fit my program'.

If you're dropping 256MB core files (the segment size in AIX), then you're essentially growing past the segment size and the -bmaxdata flag should fix the issue. I've been experiencing this recently and have not yet recompiled, but I am making many efficiency changes (true iterators have come in very handy) to combat the problem right now.

Ah a red herring - it's not hashes at all! I'm changing values in a large hash as I loop through files. The problem occurs when I come to create the key for the hash. The following code works properly on 5.6 but grows on 5.8. Delete the substitution line, and hey presto, no more leak.

use strict;

my $ret = `ps v $$`;
print "$ret\n";

for ( 1 .. 1000 ){
  my $key = ':PERSON-NUM(1:*):NI-NUM(1:*):';
  for my $bb ( "PERSON-NUM", "NI-NUM" ){
     if ( $key =~ /:$bb *\( *([0-9]+) *: *([0-9\*]+) *\) *:/ ){
        $key =~ s/:$bb *\( *[0-9]+ *: *[0-9\*]+ *\) *:/:$bb=1:/;
       }
    }
 }


$ret = `ps v $$`;
print "$ret\n";
[download]

Now I'm confused!

---

'I think the problem lies in the fact that your data doesn't fit my program'.

I ran your code with the loop iterator set 1e6, and see no memory growth under either 5.8.6 or 5.8.8.

I seem to recall that 5.8.2 (the version shown in your OP), was a particularly buggy, short-lived release. I strongly suggest you try upgrading to something newer. My memory tells me that 5.8.3 was a pretty good build as is 5.8.6 which I still use as my default install. I encountered a few problems with 5.8.8 but I think they may have been limited to the AS-Win distribution I use.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.