Fellow Monks,

Recently, I was involved in a discussion where I posted a certain Perl algorithm that I've used for quite a while for various things. I was under the impression that it was a reasonably well-designed way to tokenize a string, look up the tokens in a hash, and then replace the tokens with another one when needed. The code I have used is this:

$string =~ s/([^\s.\]\[]+)/{exists($tokens_to_match{lc $1}) ? "$tokens_to_match{lc $1}" : "$1"}/gei;

My reasoning behind this code was as follows: since hash lookup is fairly constant, using a hash lookup would provide good scalability since it avoids repeated regexp compilation. However, it was countered that this solution was not efficient at all. The person replying claimed that the performance of this algorithm is worse than O(n^2) because the way I'm using s/// is inefficient.

I am hoping that some of you can provide guidance on this problem. Is there a better way to approach this problem than my current method?

In reply to Efficient string tokenization and substitution by jpfarmer

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.