in reply to Massive Perl Memory Leak

Working code is always preferable to pseudo-code.

But...does it make any difference if you don't use Data::Dump::dump()? Eg. Just print the hash rather than dumping it. My though is that is may not be thread-safe and is accumulating internal data (say, for circular reference detection) that isn't being cleaned up or is getting replicated.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Replies are listed 'Best First'.
Re^2: Massive Perl Memory Leak
by wagnerc (Sexton) on Jun 11, 2007 at 18:08 UTC
    The hash can't be printed otherwise. It's a deep tree and just print'ing it would just give the top level keys and HASH(0x12345). Sorry I only have psuedo code. The actual program is huge[TM]. Although I could try to make a junk program that does nothing except assign to %datahash.

    The %datahash tree essentially looks like: 

    (
  "index",
  {
    baseip   => "1.2.3.4",
    deviceid => "81F1FD9964973BB63B3446881F901AF5",
    sysname  => "router",
  },
  "routes",
  {
    "0.0.0.0.0.0.0.0.2.3.4.5"         => {
                                             ifindex => 0,
                                             mask    => "0.0.0.0",
                                             net     => "0.0.0.0",
                                             nexthop => "2.3.4.5",
                                             proto   => 3,
                                             type    => 4,
                                           },
  },
  "interfaces",
  {
    1 => {
           ARP           => 3,
           ifAdminStatus => 1,
           ifAlias       => "",
           ifDescr       => "FastEthernet0/0",
           ifIndex       => 1,
           ifOperStatus  => 1,
           ifSpeed       => 100_000_000,
           ifType        => "ethernetCsmacd",
         },

    [download]
    I found the page that talked about the hash of hashes problem. It's http://www.chat11.com/Perl_Memory_Usage . Ring any bells?
[reply]
[d/l]
      he hash can't be printed otherwise. It's a deep tree and just print'ing it would just give the top level keys...

      I meant avoid the use of dump just temporarily to see if it is the cause of the memory growth.

      If avoiding it fixed the immediate problem, then you could look for another way of recursively printing deep data structures that didn't suffer the same problem. Maybe Data::Dumper, or Data::Dump::Streamer or even rmap with a little logic of your own.

      But first you need to track down the source of the problem. You could equally just comment out the print altogether.

      Is %datahash shared?

      What's in the hash doesn't matter.

      There is no logic I know of for using local rather than my in threads.

      Without sight of the full program (stick it on your scratchpad and /msg me if you don't want to post it), or a cut down but working version that demonstrates the same symptoms, it's really impossible to offer much in the way of help.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom