I added W-BER to my benchmark. It just about beats binary ('v*') on space. About the same for unpacking but slower for packing.

Any way you slice it, the single best option is to restructure the schema to cause less data to need retrieval...but then you know that :) 

## Note this run uses 1000 documents rather than 5000. 
## The relative performance remains the same though.

Run with 15000 unique words in 1000 documents (Ave: 554 words/doc)

ASCII  uses 4755001 bytes
W-BER  uses 3196726 bytes
Binary uses 3279128 bytes
Elias  uses 4118678 bytes

1 trial of Packing ascii (9.078s total)
1 trial of Unpacking ascii (2.942s total)

1 trial of Packing W-BER (16.144s total)
1 trial of Unpacking W-BER (8.219s total)

1 trial of Packing binary (9.504s total)
1 trial of Unpacking binary (8.223s total)

1 trial of Packing Elias (11.176s total)
1 trial of Unpacking Elias (9.750s total)

[download]
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

In reply to Re^6: Byte allign compression in Perl.. by BrowserUk
in thread Byte allign compression in Perl.. by MimisIVI

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.