I'm not exactly sure that the "correctness" of a refcount can be determined, as one would have to scan all memory available to Perl for references to the SV in question, and even then the memory pointing to it might not be in use anymore.

What I think should be feasible would be a (tree-structured?) dump of all memory references/SVs, together with their reference count. Then you can liberally sprinkle those calls to the dumper around and see where there is change in the tree when there should be none. On the other hand, the signal to noise ratio here is pretty bad I guess, as pretty much every op allocates or deallocates memory...

I guess that the only other recourse would be valgrind or another memory bounds checker, but I'm not sure if they (can) catch and diagnose double frees via refcount mess-ups. Maybe you can trick Perl into not reusing existing SVs, so you then get a real leak or a real access violation?

I have never done any of this, and the only thing possible without any too deep XS magic seems to me the tree dump, courtesy of Scalar::Util and PadWalker, and the output of that won't necessarily be helpfull...

While valgrind is an excellent tools to check whether perl itself has any memory related problems, it won't help at all for debugging Perl programs. valgrind sits between perl and the OS (roughly speaking) and checks memory access with respect to malloc()ed and free()d memory. valgrind doesn't know the difference between an SV and a hole in the ground.

    @a = (bless {}, 'X');
    sub X::DESTROY { @a = () }
    @a=();
__END__
[download]

Whats important to note is that the warning comes with a line number, which you can use to investigate (it should give away which variable caused it)

As I said in my orginal post I suspect there is action-at-a-distance going on. The real culprit is in one place but the bug is manifesting with respect to another variable that's getting accidently aliased.

When I run the code (which is in a module) in a standalone CGI script it doesn't give warnings. When I run the code under Apache::Registry then it starts giving warnings in many unrealated places after a few transactions.

Here is an example of one of the places where it's giving a warning.

    for my $field ( @$self{@{$_{FIELDS}}} ) {
        $field->set_related(AFTER => \@keys );
        $field->{MULTI} = 1 if  $_{MULTICODE} && ! exists $field->{MUL
+TI};
        $field->{ORDERED} = 1 if $_{ORDER} && ! exists $field->{ORDERE
+D};
        $field->{CONTEXT_KEYS} = $_{CONTEXT_KEYS} if $_{CONTEXT_KEYS} 
+&&
            ! exists $field->{CONTEXT_KEYS};
        if ( $table ) {
            $field->{TABLE} = $table unless exists $field->{TABLE};
            $field->{COLUMN} = $fieldmap->{$field} || "$field"
                unless exists $field->{COLUMN};
        }
    }

    if ( $structured_field ) { # Warning here!
        $fieldmap = {%$fieldmap};
        my @pseudo_fields = map { @$_ } values %$structured_field;
        @$fieldmap{@pseudo_fields} = map { [ $fieldmap->{$_} || $_ ] }
+ @pseudo_fields;
    }
[download]

As you see the line where the warning appears has no buisness unreferencing any scalars. The variable  $structured_field is, by the way, undef.

The perhaps interesting thing is that the variables $self and $field are not a blessed hashes but rather are objects that overload '%{}'. Furthermore the '%{}' overload on $self is done via a transitory tied hash...

  use overload
   '%{}' => sub { my $self = shift; tie my(%h), $self; \%h };

  sub TIEHASH { shift }
[download]

UPDATE Replacing the transitory tied hash with a persistant one did not fix the problem.

A cheesy but possibly effective way to solve this problem is build a custom version of Perl that allows you to log all calls to increment/decrement a particular variable. Install that in a custom location, recompile all of your XS modules against that, and see what you get.

Glancing at the code, you'd want to redefine SvREFCNT_inc and SvREFCNT_dec in sv.h. The latter is easier to add custom logic to - it calls sv_free in sv.c so you can edit that. The former is an inline macro that may be harder to edit.

From an uninformed glance at the code, if you want a custom per-scalar flag the sv_flag's struct has 8 more flags available: 0x00000010, 0x00000020, 0x00000040, 0x00000080, 0x00000001, 0x00000002, 0x00000004, and 0x00000008. However seeing how much work is going into avoiding using those makes me suspect that someone, somewhere uses them in a way that is not obvious to me.

I'd think that this would be tricky. The problem is that while you might track what Perl is doing (be careful to not follow weak references!), various XS modules are going to have pointers into Perl data structures that come from non-Perl data structures. How do you find and account for those?

I wouldn't bother. The tool can simply report those as refcount inconsitancies. The tool could show all the paths if found to each SV together with the reference count on the SV. If there are legitmate refernce paths through non-perlish structures then these should show up.

That said, when I write XS modules I avoid this problem. If I need my objects to contain hidden internal stuff that holds references to SVs I do so by hanging Perl structures off of the MAGIC list. In this way my private stuff is hidden from the Perl programmer but could still be traversed by the proposed tool.