AH! Yes. I see what you mean now. That's actually a fun problem.

My first "real world" answer would be to accept the inefficiency of the degrading accuracy of the table. But that's no fun.

If you did a true multi pass you'd run in to the problem of your universe of 8-bit symbols jumping by virtue of the previous encoding pass. So that's no fun either.

It looks to me like "given lock in to the initial approach" you'd have to take the optimum compression candidate, decrement the individual cases of its components from the frequency distribution table, including bounding characters (the spaces in the 2-grams of ' here ', for instance) for each substitution instance of 'here'. That would leave you with an accurate frequency distribution, having pulled the 40 instances of 'he', 'er', 're', ' h' and 'e ' (with the foolish assumption that 'here' always occurs in the middle of a sentence.) So the table is accurate again.

In reply to Re^5: Dynamically Updating Frequency Analysis by Voronich
in thread Dynamically Updating Frequency Analysis by Limbic~Region

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.