A couple of things I notice once I played with the expanded testcase. Your strings version is capturing single spaces somewhere. And both your strings and my regex are letting single $'s through as well.

I have a m/(..)/g version which avoids these two problems and is quicker than my split attempt, but still not as quick as your strings in it's current form.

our $RE_WORDS = qr[ (?: \$? \d+ (?:[.,] \d+ )*)+ | [\w\'!-]+ ]xo; sub tokenize_msg_w_m { my %words; @words{ shift =~ m[($RE_WORDS)]og } = (); return keys %words; }

Results

D:\Perl\test>257026 Regex: $ | want | 192.168.1.1 | $57 | confusing | yword | hopefully | combinations | LITTEL!!!!L | a | of | bit | is | This | to | will | text | this | 1,500 | Keep | out | BITH!!!!! | it | example | work | unhapp | $2.50 | little | MORE | I | some | thing | with | and | an Strings: $ | | want | 192.168.1.1 | $57 | confusing | yword | hopefully | combinations | LITTEL!!!!L | a | of | bit | is | This | to | will | text | this | 1,500 | Keep | out | BITH!!!!! | it | example | work | unhapp | $2.50 | little | MORE | I | some | thing | with | and | an Match: want | 192.168.1.1 | $57 | confusing | yword | hopefully | combinations | LITTEL!!!!L | a | of | bit | is | This | to | will | text | this | 1,500 | Keep | out | BITH!!!!! | it | example | work | unhapp | $2.50 | little | MORE | I | some | thing | with | and | an Rate regex match strings regex 526/s -- -13% -26% match 607/s 15% -- -15% strings 710/s 35% 17% --

The other question that crossed my mind was, what happens if the text contains "Do 33% of people watch T.V.?" ?


Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller

In reply to Re: Re: Re: Re: Re: tokenize plain text messages by BrowserUk
in thread tokenize plain text messages by revdiablo

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.