A closure prevents a variable from being GC'd when it goes out of scope by increasing the ref count. What I am suggesting is keeping the memory around so that you lookup the stringified reference later, the memory is still there. I am suggesting unfaking it by decreasing the ref count when you are done (no longer need to use the stringified reference).

What hash? Why use a hash and how would that save memory if you do the Tie::RefHash thing of storing the smashed and unsmashed references?

In my original post, I indicated I was trying to change a HoA into a regular hash. The value would be a fixed number of bytes, but unfortunately part of the packed string would need to be a reference. What I am saying is that as long as you control the addition/deletion/modification of this hash, you can also properly control the GC.

If you do the RefHash thing, you don't need to do any controlling--once you delete the (value) reference, Perl takes care of the rest. But trading a hash for an array is not going to save memory.

Not my goal. See my original problem. The hash keys will be regular plain jane keys - it is the values that will be conserving memory. Instead of being anonymous arrays they would be packed strings.

A closure prevents a variable from being GC'd when it goes out of scope by increasing the ref count.

I know what closures are and how they work. I'm asking how you are going to fake a closure?

I was trying to change a HoA into a regular hash ...

And I was ignoring the outer level of your individual problem and concentrating upon the core detail of how to store references in a packed string 'array'. As soon as you smash the reference, Perl no longer has any knowledge that you have a reference, so if the thing the caller passed to you that you have taken (and smashed) a reference to, goes out of scope in the callers code, Perl will feel free to GC it.

To avoid that, you would need to keep a copy of the unsmashed reference around. To do that, you need to put it somewhere:

• You could put it into an array, but then when it comes to deleting it, you need to grep the entire array.
• Or you could put it into a hash (which is what I thought you were suggesting).

  The problems with that are.

  1. You need to store it as both the key and the value. The key so that you can look it up quickly when you come to delete it. The value so that perl retains knowledge of and manages the reference count on the referrent.
  2. Using a hash to store the values of an array (as well as storing them in a packed string, is a cure worse than the desease. You are now storing 3 copies of the reference + the referrent and the overhead of a hash is far higher than the array that you are trying to save memory for.

Not my goal. ... ... ...

The Ho... bit doesn't matter. It could equally be an AoAs or an HoAoHoAs etc.

The main part of your original problem description is how to replace an array with a packed string, regardless of where the array/packed string will itself be stored.

And, as you pointed out, the problem with packing an array, is that it only really works with numeric values.

Storing the address of a reference to the actual data item neatly deals with that--except now how do you ensure the survival of the referred to item as you only have a smashed reference to it, and perl doesn't know you have it? See above.

I get the feeling that we are talking past each other here, but I will say that I've given this a lot of thought and code over the past two years, and I haven't discovered the magic ingredient yet. If you do, I'll be the first to congratulate you.

However, that none of the prolific p5p guys has implemented a complete compact array solution yet, given their great knowledge of the internals, makes me feel a little better about not having found the solution. I probably would have coded my germ of an idea by now, were it not that I feel that this can only be done by writing XS code--possibly including a patch to the core--and I find doing anything at that level, an exercise in total frustration. There are obviously good techniques for developing at that level, I'm just not party to them; haven't found the roadmap to doing it; and don't see any great willingness on the behalf of those in the know, to sharing their knowledge.

---

Examine what is said, not who speaks.

Silence betokens consent.

Love the truth but pardon error.

BrowserUk,
I know what closures are and how they work. I'm asking how you are going to fake a closure?

Hmmm. Classic example of how things that are clear in your mind don't make any sense to anyone else because you assume they know what you are thinking. By faking the closure, I mean get the same effect - causing the reference count to increase by one so when it goes out of scope it doesn't reach 0.

To avoid that, you would need to keep a copy of the unsmashed reference around. To do that, you need to put it somewhere:

This is the exact thing I am proposing you don't have to do. It is in scope at the time you are stringifying it. If the only thing (which is my understanding) that is keeping it from being GC'd when it goes out of scope is the reference count - just increase it. You use the stringified address to look it up later, you GC it by lowering the ref count.

If the idea isn't clear (forget workable for now) then let me know and I will try a non-commentary approach.

Cheers - L~R

P.S. I didn't indicate that XS wouldn't be required though it would be really cool if it wasn't.