That's due to the memory policies of Perl. You undef a large
hash, which means that for each value stored, Perl has to
decrement the refcount of the value, check to see if it's
zero, and if it's zero, free its memory (and possibly decrement other ref counters). Finally, it needs to free up
the memory structure of the hash itself, which will be lots
of linked lists, so there are lots of little pieces of memory
to be freed.
Abigail | [reply] |
Hi Abigail!
How about clearing all key/value pairs with:
%myhash = ();
How does this work internally?
Thanks, Michele.
| [reply]
[d/l] |
It works the same way. The killer is freeing up all the bucket entries, and when you undef the hash they get freed just the same as when the hash goes out of scope. There's rather a lot of small memory malloc'ing going on (with corresponding extra overhead in memory footprint that's generally unaccounted for) and when those get freed it triggers some pathological behaviour in some versions of glibc's memory system.
| [reply] |
We had the same problem - traced to a buggy version of GCC - our production version is buggy gcc 2.95.4
The solution was to
1) Use a GLOBAL scope for the offending large hash - I know it runs against the grain, but you DONT want it to go out of scope or your process will sit around for ages running buggy malloc() reshuffles.
2) Use POSIX::_exit($exitval); instead of exit() - this exits immediately without bothering to do the last garbage collection
For us, with hashes of 200MB - 1.2GB this reduces task times from 35 minutes to 5 minutes!
Regards
Jeff | [reply] |
Well, Perl does have it's own malloc, which you can turn on
if you compile Perl with the -Dusemymalloc.
Abigail
| [reply] |
We tried Perls malloc - even worse problems than the GCC 2.95 malloc, which runs slow, but at least runs reliably to completion.
We also tried upgrading GCC but then our MySQL connections would fail after several 10s of MB interaction with the MySQL server.
regards,
Jeff
| [reply] |
If you're only worried about the memory, don't be. If you're not using it, it'll go to swap space soon enough. perl won't actually free that space to the system until the interpreter exists, anyway.
----
I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
-- Schemer
Note: All code is untested, unless otherwise stated
| [reply] |