in reply to Re: Re: Hitting memory limit? (ActiveState Perl)
in thread Hitting memory limit? (ActiveState Perl)
Your Total Commited Charge far exceeds your Total Physical Memory--by about 50%. You've been into swapping for a considerable time.
Although the TCC is fairly static, suggesting that the hash isn't growing much, each time you access a key within the existing hash, it's quite possible that that perl would have cycle through the entire hash structure to locate the next value, which in turn could mean the OS having to swap a huge portion of the process' image.
You probably had trouble hearing your mp3 over the sound of the disc thrashing--your nieghbour's probably had the same problem:)
Perls hashes are pretty memory hungry, and if you are nesting them, filling memory doesn't take too much effort. I induced this machine, with 512MB ram into swapping in 80 seconds with a hash containing a little under 6_000_000 keys with empty (undef) values. If your values are only simple scalars you'll get there much quicker. If they are themselves hashes or arrays, much quicker still.
sub rndStr{ join '', @_[ map{ rand @_ } 0 .. shift ] }; $! = 1; ( $_ % 1000 or printf( "\r$_ : %s ", times ) ) and $h{ rndStr 8, 'a'..'z', 'A' .. 'Z', 0 .. 9 } = undef for 0 .. 10_000_000; 5858000 : 80.703
You could find out how many keys it takes to induce swapping by disabling swapping and logging scalar keys %your_main_hash periodically as you fill it. If the last number displayed before you get perls "Out of memory" error is not to far short of your expected final size, then it might be worth looking at seeing how you could use less memory. If it is a long way short, then that effort is almost certainly not worth it.
There are some alternatives to hashes which can save memory depending on the nature of your keys and what use you are making of the hash (see A (memory) poor man's <strike>hash</strike> lookup table. for one possible, part solution), but if you are regularly dealing with these volumes of data, a DB of some form is probably your easiest option.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re: Re: Re: Re: Hitting memory limit? (ActiveState Perl)
by Anonymous Monk on Jan 22, 2004 at 13:30 UTC | |
|
Re: Re: Re: Re: Hitting memory limit? (ActiveState Perl)
by NYTAllergy (Initiate) on Jan 22, 2004 at 14:22 UTC | |
by BrowserUk (Patriarch) on Jan 23, 2004 at 01:45 UTC |