A closure prevents a variable from being GC'd when it goes out of scope by increasing the ref count.

I know what closures are and how they work. I'm asking how you are going to fake a closure?

I was trying to change a HoA into a regular hash ...

And I was ignoring the outer level of your individual problem and concentrating upon the core detail of how to store references in a packed string 'array'. As soon as you smash the reference, Perl no longer has any knowledge that you have a reference, so if the thing the caller passed to you that you have taken (and smashed) a reference to, goes out of scope in the callers code, Perl will feel free to GC it.

To avoid that, you would need to keep a copy of the unsmashed reference around. To do that, you need to put it somewhere:

• You could put it into an array, but then when it comes to deleting it, you need to grep the entire array.
• Or you could put it into a hash (which is what I thought you were suggesting).

  The problems with that are.

  1. You need to store it as both the key and the value. The key so that you can look it up quickly when you come to delete it. The value so that perl retains knowledge of and manages the reference count on the referrent.
  2. Using a hash to store the values of an array (as well as storing them in a packed string, is a cure worse than the desease. You are now storing 3 copies of the reference + the referrent and the overhead of a hash is far higher than the array that you are trying to save memory for.

Not my goal. ... ... ...

The Ho... bit doesn't matter. It could equally be an AoAs or an HoAoHoAs etc.

The main part of your original problem description is how to replace an array with a packed string, regardless of where the array/packed string will itself be stored.

And, as you pointed out, the problem with packing an array, is that it only really works with numeric values.

Storing the address of a reference to the actual data item neatly deals with that--except now how do you ensure the survival of the referred to item as you only have a smashed reference to it, and perl doesn't know you have it? See above.

I get the feeling that we are talking past each other here, but I will say that I've given this a lot of thought and code over the past two years, and I haven't discovered the magic ingredient yet. If you do, I'll be the first to congratulate you.

However, that none of the prolific p5p guys has implemented a complete compact array solution yet, given their great knowledge of the internals, makes me feel a little better about not having found the solution. I probably would have coded my germ of an idea by now, were it not that I feel that this can only be done by writing XS code--possibly including a patch to the core--and I find doing anything at that level, an exercise in total frustration. There are obviously good techniques for developing at that level, I'm just not party to them; haven't found the roadmap to doing it; and don't see any great willingness on the behalf of those in the know, to sharing their knowledge.

BrowserUk,
I know what closures are and how they work. I'm asking how you are going to fake a closure?

Hmmm. Classic example of how things that are clear in your mind don't make any sense to anyone else because you assume they know what you are thinking. By faking the closure, I mean get the same effect - causing the reference count to increase by one so when it goes out of scope it doesn't reach 0.

To avoid that, you would need to keep a copy of the unsmashed reference around. To do that, you need to put it somewhere:

This is the exact thing I am proposing you don't have to do. It is in scope at the time you are stringifying it. If the only thing (which is my understanding) that is keeping it from being GC'd when it goes out of scope is the reference count - just increase it. You use the stringified address to look it up later, you GC it by lowering the ref count.

If the idea isn't clear (forget workable for now) then let me know and I will try a non-commentary approach.

Cheers - L~R

P.S. I didn't indicate that XS wouldn't be required though it would be really cool if it wasn't.

So, forgetting all the stuff about faking closures and stuff, your proposing to "just increase the refcount"?

Assuming that you can do that, what happens if the thing you just increased the refcount of, was previously deliberatly weakened by the caller for inclusion into self-referential structure? Or after they've given it to you?

How does it play if your 'array' is initialised by

tie my @a, 'Compact::Array';

@a = ( 'default' ) x 1_000_000;
[download]

The refcount on the SV pointing at that constant string 'default' is now 1000000. That's a million chances for it never getting cleaned up properly.

I'm not saying that this isn't possible to do--it is--but these are all things I have had problems with.

It gets even more complicated when you start thinking of the effect of storing references to shared objects and how that plays with threads and threads::shared.

I've spent a fair amount of time looking at the code in threads.xs & shared.xs that has to play similar games with reference counts.

The guys that wrote that code are a darn sight cleverer than I, and still those modules have probably been the source of more leakage than amost any other. That tells me that it's not trivial, even if my failure to make it work properly is down to my stupidity.

In each case, there are a subset of situations in which any given solution will work fine, and for a given specific task, they work okay, but for a general purpose solution, you need to consider all these possibilities and more.

---

Examine what is said, not who speaks.

Silence betokens consent.

Love the truth but pardon error.