Memory Leak Upon Cloning an XS tied hash

gmarler has asked for the wisdom of the Perl Monks concerning the following question:

In this question, I'll be referring to the XS module found here, particularly the kstat.xs file contained within:

https://github.com/gmarler/Solaris-kstat

This module creates a 3 level tied hash in XS, of the form:

$k->{$module}->{$instance}->{$statname}
[download]

For the sake of this discussion, I'm looking at the data for CPU 0:

$k->{cpu}->{0}->{sys}
[download]

The way the module is supposed to work is like so:

The first time you reference a key in the hash, its value is dynamically pulled from the libkstat API and populated, then marked read-only.
From that moment on, that data is supposed to be frozen, until you explicitly call the ->update() method on the object, which iterates over it in XS code, and updates any of the read-only items the hash has already been populated with from libkstat.

So, it's often the case that you'll want to take a measurement, wait a time interval, take another measurement, then compare the two. This has turned out to be surprisingly hard to accomplish, due to the following:

If you try to have more than one Solaris::kstat XS object, they seem to interact somehow, causing a memory leak; looking at refcounts shows certain ones below the top 3 tied hashes have refcounts of 2 instead of just 1. Using just one object doesn't exhibit this behavior.
Trying to clone the object (it is supposed to remain frozen, after all, until the next update()) yields surprising results.

Using Clone::clone causes some of the underlying items to change or be re-read from libkstat; there's no real pattern here, so I'm at a loss. Here's an example test:

use Test::Most;
use Test::Refcount;

use Solaris::kstat;
use Clone        qw();

my $k = Solaris::kstat->new();

# 'Freeze' kstat data for cpu 0:
ok( defined($k->{'cpu'}->{0}->{sys}), 'CPU kstat should be defined');

# Clone the data
my $c1 = Clone::clone($k);

# Compare the keys/values under the above, as they should remain the s
+ame
foreach my $key (keys %{$k->{cpu}->{0}->{sys}}) {
  eq_or_diff($c1->{cpu}->{0}->{sys}->{$key}, $k->{cpu}->{0}->{sys}->{$
+key},
             "Clone::clone cpu:0:sys:$key should be identical");
}

done_testing();
[download]

Which produces the unexpected:

$ prove -v t/clone.t
t/clone.t ..
ok 1 - CPU kstat should be defined
ok 2 - CPU 0 comparison should be identical
ok 3 - Clone::clone cpu:0:sys:cpu_ticks_kernel should be identical
ok 4 - Clone::clone cpu:0:sys:modunload should be identical
not ok 5 - Clone::clone cpu:0:sys:cpu_nsec_kernel should be identical

#   Failed test 'Clone::clone cpu:0:sys:cpu_nsec_kernel should be iden
+tical'
#   at t/clone.t line 20.
# +---+-----------------+-----------------+
# | Ln|Got              |Expected         |
# +---+-----------------+-----------------+
# *  1|189890471970670  |189890471964170  *
# +---+-----------------+-----------------+
ok 6 - Clone::clone cpu:0:sys:rcvint should be identical
ok 7 - Clone::clone cpu:0:sys:bwrite should be identical
ok 8 - Clone::clone cpu:0:sys:namei should be identical
not ok 9 - Clone::clone cpu:0:sys:xcalls should be identical

#   Failed test 'Clone::clone cpu:0:sys:xcalls should be identical'
#   at t/clone.t line 20.
# +---+-------------+-------------+
# | Ln|Got          |Expected     |
# +---+-------------+-------------+
# *  1|12046262714  |12046262711  *
# +---+-------------+-------------+
ok 10 - Clone::clone cpu:0:sys:rw_rdfails should be identical
ok 11 - Clone::clone cpu:0:sys:mdmint should be identical
ok 12 - Clone::clone cpu:0:sys:intrunpin should be identical
ok 13 - Clone::clone cpu:0:sys:sysread should be identical
ok 14 - Clone::clone cpu:0:sys:cpu_load_intr should be identical
ok 15 - Clone::clone cpu:0:sys:bread should be identical
ok 16 - Clone::clone cpu:0:sys:syswrite should be identical
ok 17 - Clone::clone cpu:0:sys:canch should be identical
ok 18 - Clone::clone cpu:0:sys:ufsipage should be identical
ok 19 - Clone::clone cpu:0:sys:pswitch should be identical
ok 20 - Clone::clone cpu:0:sys:intrthread should be identical
ok 21 - Clone::clone cpu:0:sys:phread should be identical
ok 22 - Clone::clone cpu:0:sys:cpu_ticks_wait should be identical
ok 23 - Clone::clone cpu:0:sys:wait_ticks_io should be identical
ok 24 - Clone::clone cpu:0:sys:ufsinopage should be identical
ok 25 - Clone::clone cpu:0:sys:ufsdirblk should be identical
ok 26 - Clone::clone cpu:0:sys:idlethread should be identical
ok 27 - Clone::clone cpu:0:sys:phwrite should be identical
not ok 28 - Clone::clone cpu:0:sys:cpu_nsec_idle should be identical
  
#   Failed test 'Clone::clone cpu:0:sys:cpu_nsec_idle should be identi
+cal'
#   at t/clone.t line 20.
# +---+------------------+------------------+
# | Ln|Got               |Expected          |
# +---+------------------+------------------+
# *  1|6135297234064540  |6135297233813960  *
# +---+------------------+------------------+
not ok 29 - Clone::clone cpu:0:sys:cpu_ticks_idle should be identical
  
#   Failed test 'Clone::clone cpu:0:sys:cpu_ticks_idle should be ident
+ical'
#   at t/clone.t line 20.
# +---+------------+------------+
# | Ln|Got         |Expected    |
# +---+------------+------------+
# *  1|6135297234  |6135297233  *
# +---+------------+------------+
ok 30 - Clone::clone cpu:0:sys:procovf should be identical
ok 31 - Clone::clone cpu:0:sys:nthreads should be identical
ok 32 - Clone::clone cpu:0:sys:intrblk should be identical
ok 33 - Clone::clone cpu:0:sys:rawch should be identical
ok 34 - Clone::clone cpu:0:sys:ufsiget should be identical
ok 35 - Clone::clone cpu:0:sys:iowait should be identical
ok 36 - Clone::clone cpu:0:sys:cpumigrate should be identical
not ok 37 - Clone::clone cpu:0:sys:snaptime should be identical

#   Failed test 'Clone::clone cpu:0:sys:snaptime should be identical'
#   at t/clone.t line 20.
# +---+------------------+------------------+
# | Ln|Got               |Expected          |
# +---+------------------+------------------+
# *  1|9606917259190740  |9606917258937140  *
# +---+------------------+------------------+
ok 38 - Clone::clone cpu:0:sys:writech should be identical
ok 39 - Clone::clone cpu:0:sys:cpu_ticks_user should be identical
ok 40 - Clone::clone cpu:0:sys:crtime should be identical
ok 41 - Clone::clone cpu:0:sys:sysvfork should be identical
ok 42 - Clone::clone cpu:0:sys:sema should be identical
ok 43 - Clone::clone cpu:0:sys:sysfork should be identical
not ok 44 - Clone::clone cpu:0:sys:intr should be identical

#   Failed test 'Clone::clone cpu:0:sys:intr should be identical'
#   at t/clone.t line 20.
# +---+-------------+-------------+
# | Ln|Got          |Expected     |
# +---+-------------+-------------+
# *  1|16979446882  |16979446880  *
# +---+-------------+-------------+
ok 45 - Clone::clone cpu:0:sys:modload should be identical
ok 46 - Clone::clone cpu:0:sys:rw_wrfails should be identical
ok 47 - Clone::clone cpu:0:sys:xmtint should be identical
ok 48 - Clone::clone cpu:0:sys:readch should be identical
ok 49 - Clone::clone cpu:0:sys:cpu_nsec_intr should be identical
ok 50 - Clone::clone cpu:0:sys:lwrite should be identical
ok 51 - Clone::clone cpu:0:sys:outch should be identical
ok 52 - Clone::clone cpu:0:sys:mutex_adenters should be identical
ok 53 - Clone::clone cpu:0:sys:cpu_nsec_stolen should be identical
ok 54 - Clone::clone cpu:0:sys:sysexec should be identical
ok 55 - Clone::clone cpu:0:sys:inv_swtch should be identical
ok 56 - Clone::clone cpu:0:sys:syscall should be identical
ok 57 - Clone::clone cpu:0:sys:trap should be identical
ok 58 - Clone::clone cpu:0:sys:bawrite should be identical
ok 59 - Clone::clone cpu:0:sys:cpu_ticks_stolen should be identical
ok 60 - Clone::clone cpu:0:sys:lread should be identical
ok 61 - Clone::clone cpu:0:sys:class should be identical
ok 62 - Clone::clone cpu:0:sys:msg should be identical
ok 63 - Clone::clone cpu:0:sys:cpu_nsec_user should be identical
[download]

Using Data::Clone::clone instead yields this test result:

Modification of a read-only value attempted at t/clone.t line 24.
[download]

And using Storable::dclone yields this behavior:

Solaris::kstat: read_kstats: lost ~ magic at /usr/perl5/site_perl/5.22
+.0/Test/Differences.pm line 532.
[download]

So, I'm wondering if anyone can help me with any of:

Determining what has been overlooked, or incorrectly implemented, in the XS hash tie implementation in the module that leads to this unusual behavior
Am I going to have to implement a clone() sort of method directly in the XS module? If so, where can I find a starting point for learning how this works?
Is there a much simpler way to accomplish the ultimate goal of just having a dumb hash copy of the "last" reading of the kstat of interest, so that when I run $k->update(), I have something to compare against?

Many thanks for any feedback.

Comment on Memory Leak Upon Cloning an XS tied hash Select or Download Code

Replies are listed 'Best First'.
Re: Memory Leak Upon Cloning an XS tied hash by Anonymous Monk on Sep 21, 2015 at 20:51 UTC
Here is an old gem of a tip Do it in perl :) Forget about XS and simply write as much as possible in perl, write a pure perl tied hash that calls various XS methods to populate the hash and cache stuff writing a tied interface in XS is ~~kinda zany~~ ambitious	[reply]