brian_d_foy has asked for the wisdom of the Perl Monks concerning the following question:

Here's an interesting problem, and maybe someone can educate me on what's going on with it. I suspect that Devel::Size has a problem with tied objects but I couldn't find anyone talking about it. While investigating some stuff for Mastering Perl, I wanted to see how much memory I could save (trading for speed) by packing an array into a scalar, like Tie::Array::PackedC does. If I'm not using an array, I shouldn't have to incur the scalar variable overhead for every element, which could be 20 bytes or so.

I noticed that calling Devel::Size::total_size() on a tied array (haven't tried the other structures) returns a number that seems singularly dependent on that value that FETCHSIZE returns. It took me a while to suss this out, but I finally created this mostly empty class. The FETCHSIZE method simply returns the value I passed to TIEARRAY.

(If you want to play with this, I've put up a tarball on my Mastering Perl site: Tie-Array-Null-0.01.tar.gz) 

package Tie::Array::Null;
# $Id: Cycle.pm,v 1.12 2005/03/08 22:29:30 comdog Exp $
use strict;
use base qw(Tie::Array);

use vars qw( $VERSION );
$VERSION = 0.01;

sub TIEARRAY  { bless \ $_[1], $_[0] }
sub FETCHSIZE { ${ $_[0] } }
sub STORE     { undef }
sub FETCH     {    undef }
sub STORESIZE {   }
sub EXTEND    {   }
sub CLEAR     {   }
sub EXISTS    { 0 }
sub DESTROY   {   }
    
"Tie::Array::Null";

[download]

The module doesn't store anything, and I don't expect the tied array size to grow. Devel::Size::total_size( @tied_array ), I think, should report something small and constant no matter what I do. I wrote a test script to inspect the sizes as I add elements to the array. I suspect that Devel::Size is counting that same thing multiple times, but my internals Kung-Fu isn't strong enough to point out just where that's happening. 

#!/usr/bin/perl
# $Id$
use strict;

use Test::More 'no_plan';

use Devel::Size qw(total_size); 
use Tie::Array::Null;

foreach my $size ( 0, 1, 5, 10, 100 )
    {
    report( $size )
    }

pass();

sub report {
    my $size = shift;

    my $object = tie my( @tied_array ), 'Tie::Array::Null', $size;
    isa_ok( $object, 'Tie::Array::Null' );
    
    my $tied_array_size = total_size( \@tied_array );
    
    my @regular_array = ();

    print STDERR "\n----Testing for size $size-----\n";
    print STDERR "Tied array size before is " . 
        total_size( \@tied_array ) . "\n";
    print STDERR "Regular array size before is " . 
        total_size( \@regular_array ) . "\n";
    
    
    print STDERR "-" x 73, "\n";
    printf STDERR "%5s  %5s  %5s  %5s\n", qw(iter tied len reg);
    
    for( my $i = 0; $i < 1000; $i++ )
        {
        $tied_array[$i] = 'T';
        push @regular_array, 'T';
        unless( $i % 100 )
            {
            printf STDERR "%5d  %5d  %5d  %5d\n",
                $i,
                total_size( \@tied_array ),
                total_size( \$object ),
                total_size( \@regular_array );
            }
        }
    }

[download]

Once you run that code, change the number in FETCHSIZE and run the test script again. The number Devel::Size::total_size( @tied_array ) changes, roughly according to some offset along with a fixed number multiplied by the number FETCHSIZE returns (and that fixed number probably matters on your platform and particular compilation options). The actual object size grows as I expect (according to what it stores), but I don't beleive the numbers for the "tied" column. 

----Testing for size 0-----
Tied array size before is 108
Regular array size before is 56
--------------------------------------------------------------------
 iter   tied    obj    reg
    0    108     62     94
  100    108     62   3174
  200    108     62   6286
  300    108     62   9910
  400    108     62  12510
  500    108     62  15110
  600    108     62  19758
  700    108     62  22358
  800    108     62  24958
  900    108     62  27558

----Testing for size 1-----
Tied array size before is 220
Regular array size before is 4132
--------------------------------------------------------------------
 iter   tied    obj    reg
    0    220     62   4158
  100    220     62   6758
  200    220     62   9358
  300    220     62  11958
  400    220     62  14558
  500    220     62  17158
  600    220     62  19758
  700    220     62  22358
  800    220     62  24958
  900    220     62  27558

----Testing for size 5-----
Tied array size before is 556
Regular array size before is 4132
-------------------------------------------------------------------
 iter   tied    obj    reg
    0    556     62   4158
  100    556     62   6758
  200    556     62   9358
  300    556     62  11958
  400    556     62  14558
  500    556     62  17158
  600    556     62  19758
  700    556     62  22358
  800    556     62  24958
  900    556     62  27558

----Testing for size 10-----
Tied array size before is 976
Regular array size before is 4132
--------------------------------------------------------------------
 iter   tied    obj    reg
    0    976     63   4158
  100    976     63   6758
  200    976     63   9358
  300    976     63  11958
  400    976     63  14558
  500    976     63  17158
  600    976     63  19758
  700    976     63  22358
  800    976     63  24958
  900    976     63  27558

----Testing for size 100-----
Tied array size before is 8536
Regular array size before is 4132
--------------------------------------------------------------------
 iter   tied    obj    reg
    0   8536     64   4158
  100   8536     64   6758
  200   8536     64   9358
  300   8536     64  11958
  400   8536     64  14558
  500   8536     64  17158
  600   8536     64  19758
  700   8536     64  22358
  800   8536     64  24958
  900   8536     64  27558

[download]

Curiously, once I fix a few bugs in Tie::Array::PackedC's STORE method (so it correctly updates the count and extends the "array" when necessary), it has this problem too. Indeed, it wasn't until I realized that the FETCHSIZE method from that module returned the wrong value that I was led to the idea that this problem has something to do with how FETCHSIZE and Devel::Size interact.

--
brian d foy <brian@stonehenge.com>
Subscribe to The Perl Review

Replies are listed 'Best First'.
Re: Should Devel::Size work on tied arrays?
by martin (Friar) on Feb 01, 2006 at 16:08 UTC
    Kung-Fu is a good description of the nature of what Devel::Size has to do in order to inspect perl data internals. I agree that you identified a limitation not obvious from the module's documentation.

    On top of my wish list for things to be able to get the size of would be closures.

    I wonder if we can expect a tool like Devel::Size to become much more complete, given how perl's intestines are not only eclectic to boot but also a moving target.

    Another approach to track memory consumption might be called for. Perhaps something along the lines of the black-box attitude of Benchmark, only measuring space rather than time.

    Any Takers?

[reply]
[reply]
        maybe, something like Memchmark?

        Almost, but not quite, since Memchmark watches behaviour of code while the point here is to examine static data.

        A helpful technique might be to duplicate data over and over in order to increase the scale (though that is not trivial, think of shared hash keys and deeply nested references) while tracking overall memory consumption, and then doing some math.

[reply]
      Hrm. Devel::Size should handle closures properly -- I remember adding in code last summer to walk pads attached to code references, and some testing showed that it looked to work, but I may well have missed something in there. I'll go look and see what's going on.

      And yeah, Devel::Size does some... interesting things to get the total size of data structures. There's lots of evil knowledge of the size of things. For the most part that knowledge is version-independent, but not all of it. (Arguably there should be a Devel::Size-ish thing bundled as part of a perl distribution because it is so potentially version-dependent, but that's throwing more work on the core internals group, and they've enough to do as it is)

[reply]
        Devel::Size should handle closures properly

        Uh-oh. As coderefs are mentioned in the bugs section it didn't occur to me to file a bug report. Devel::Size appears to report unreasonably constant sizes of closures, and I get "Attempt to free unreferenced scalar" warnings also. Please /msg me if you need more details.

[reply]
Re: Should Devel::Size work on tied arrays?
by Elian (Parson) on Feb 01, 2006 at 20:34 UTC
    Devel::Size should work on tied anything -- there's code in it to detect attached magic structures and extract their sizes. I thought I had the code in there such that it wouldn't trigger the magic when walking the internal data structures, but I may have bobbled that. Devel::Size, unfortunately, doesn't have a particularly comprehensive test suite. (In part because the size of things is so terribly dependent on architecture and in part because I couldn't figure out a good way to test it)

    I'll grab this code and start beating on Devel::Size (which needs some beating anyway, as there's a bug in 0.64 that needs fixing) and see what I can see.

[reply]

Back to Seekers of Perl Wisdom