Well, maybe another example would be clearer: To correctly tokenize the code "if (expr) /...", you have to know that "if" is a keyword. Otherwise, it looks like a function call followed by a division operator, rather than an if statement that includes a regular expression. And then there's automatic semicolon insertion, which has to happen when the higher-level syntax doesn't allow the two tokens surrounding a linefeed to be consecutive.

I'm not a parsing expert (had to figure it out myself), so I'm happy to hear of any better approaches. Any solution has to be fast.

In any case, I'm still looking for an alternative to setting pos() to allow consecutive zero-length matches on UTF-8 strings. Hoping not to rewrite the whole routine to avoid zero-length matches.

In reply to Re^4: Is there a way to allow consecutive zero-length matches without using pos()? by jsm
in thread Is there a way to allow consecutive zero-length matches without using pos()? by jsm

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.