in reply to Techniques On Saving Memory

Take a look at Devel::Pointer. It has routines for unsmashing smashed references. It works, and I have used it to do something very similar to what you are attempting.

One problem is that you also need to store the type of ref (scalar/array/hash/code) along with the smashed address. That screws the otherwise convenient 32-bit / element storage requirement making the math messier than it ought to be.

The second is you also need to store unsmashed copies of the references so that perl will not GC the elements to which you are storing smashed refs.

That duplication of storage negates much of the benefit, combined with the fact that you are now responsible for managing the memory instead of allowing Perl to take care of it, makes for a heavier and riskier module than might otherwise be the case.

I have the germs of an idea for dealing with that also, but the tools are not available to me to implement it (yet).

Examine what is said, not who speaks.
Silence betokens consent.
Love the truth but pardon error.

Replies are listed 'Best First'.
Re^2: Techniques On Saving Memory
by halley (Prior) on Mar 09, 2005 at 19:05 UTC
    you also need to store unsmashed copies of the references so that perl will not GC the elements to which you are storing smashed refs

    Ooo, ouch. Good point.

    I think you will find, case after case, that by the time you're done "improving" a perl feature, you're going to have a solution that is as complex (in memory and time and code and shortcomings) as the perl implementation itself.

    --
    [ e d @ h a l l e y . c c ]

[reply]

      I've been trying to get this idea right for almost two years now. On two occasions I've even reached the point where I thought I was ready to let code loose on the world, but then I discovered something which I wasn't able to handle without duplicating the references in a hash, which completely negates any benefit at all.

      As I said, I have the germ of a solution, but as I am entirely uncomfortable in using XS, or even XS modules in ways that move beyond their authors intent, it will remain just an idea until I can think of a way of proving it's safety.

      Examine what is said, not who speaks.
      Silence betokens consent.
      Love the truth but pardon error.
[reply]
Re^2: Techniques On Saving Memory
by Limbic~Region (Chancellor) on Mar 09, 2005 at 19:21 UTC
    BrowserUk,
    The second is you also need to store unsmashed copies of the references so that perl will not GC the elements to which you are storing smashed refs. That duplication of storage negates much of the benefit,...

    tye was quick to point out an easy solution for the problem was to use a parallel hash as a lookup table. I admitted it was a good straight forward answer but defeated my purpose. On the other hand if an anonymous sub can be made to take up less space than the anonymous array: 

    sub compact {
    my $ref = shift;
    my $pck;  # Packed information except reference

    # purposefully avoiding dispatch table
    # This keeps us at HoC
    return sub {
        if    ( $_[0] eq 'data' ) { return $ref };
        elsif ( $_[0] eq 'blah' ) { return unpack('A1', $pck) }
        # where each unpack format gets desired data
    };
}

$hash{foo} = compact( [1..10] );
$hash{foo}->( 'data' )[3] = 42;

    [download]

    The interface could use work and it doesn't really explore any uncharted territory for me. It also is still not a single hash. We are just trading anonymous subs for anonymous arrays.

    Cheers - L~R

[reply]
[d/l]
Re^2: Techniques On Saving Memory
by japhy (Canon) on Mar 09, 2005 at 19:19 UTC
    My module (now v0.02!) doesn't seem to suffer from the garbage collection problem, and it doesn't require you tell it what kind of reference you expect back. It's not pure Perl (unlike Devel::Pointer::PP), but it is easier to use.

    (Update: I misinterpreted the garbage collection issue.)

    _____________________________________________________
    Jeff japhy Pinyan, P.L., P.M., P.O.D, X.S.: Perl, regex, and perl hacker
    How can we ever be the sold short or the cheated, we who for every service have long ago been overpaid? ~~ Meister Eckhart
[reply]
Re^2: Techniques On Saving Memory
by Limbic~Region (Chancellor) on Mar 09, 2005 at 21:03 UTC
    BrowserUk,
    Thinking out loud here with only the most elementary understanding of Perl internals - why can't you fake a closure? Basically increase the ref count without actually making the closure. You GC it by decreasing it. As long as you control the creation/modification/deletion of the hash keys and their values, I don't see an obvious problem?

    Cheers - L~R

[reply]
      ... why can't you fake a closure?...

      Sorry, but I don't even begin to see how to fake a closure? And how would you unfake it?

      As long as you control the creation/modification/deletion of the hash keys and their values, ...

      What hash? Why use a hash and how would that save memory if you do the Tie::RefHash thing of storing the smashed and unsmashed references?

      If you do the RefHash thing, you don't need to do any controlling--once you delete the (value) reference, Perl takes care of the rest. But trading a hash for an array is not going to save memory.

      Examine what is said, not who speaks.
      Silence betokens consent.
      Love the truth but pardon error.
[reply]
        BrowserUk,
        Sorry, but I don't even begin to see how to fake a closure? And how would you unfake it?

        A closure prevents a variable from being GC'd when it goes out of scope by increasing the ref count. What I am suggesting is keeping the memory around so that you lookup the stringified reference later, the memory is still there. I am suggesting unfaking it by decreasing the ref count when you are done (no longer need to use the stringified reference).

        What hash? Why use a hash and how would that save memory if you do the Tie::RefHash thing of storing the smashed and unsmashed references?

        In my original post, I indicated I was trying to change a HoA into a regular hash. The value would be a fixed number of bytes, but unfortunately part of the packed string would need to be a reference. What I am saying is that as long as you control the addition/deletion/modification of this hash, you can also properly control the GC.

        If you do the RefHash thing, you don't need to do any controlling--once you delete the (value) reference, Perl takes care of the rest. But trading a hash for an array is not going to save memory.

        Not my goal. See my original problem. The hash keys will be regular plain jane keys - it is the values that will be conserving memory. Instead of being anonymous arrays they would be packed strings.

        Cheers - L~R

[reply]
[d/l]
Re^2: Techniques On Saving Memory
by diotalevi (Canon) on Mar 09, 2005 at 23:32 UTC
[reply]

In Section Seekers of Perl Wisdom