Howdy Monks,

How can I tell if a regular expression is matching but fails merely because it ran out of data?

For instance, a lexer might have tokens that can cross line boundaries such as strings and comments. Not wanting to slurp the file in one go so as to minimize memory footprint, this lexer has a line buffer which it keeps tidy, getting more lines as needed and dereferencing lines when no longer needed. In such a case, it is possible that a regex match might start but not finish before needing more data in the line buffer. In such a case, I'd want to read in another line and try the match again or, somehow, continue the match. In either case I need to differentiate between the match failing due to bad content versus failing due to lack of content.

Any ideas?

Update...

Thank you for the good ideas. I am going to look into using the (?=\Z(?{ somecode }))? pattern for this. I agree that false positive, incomplete matches are also a problem. I think a bit of code to modify the pattern, automatically inserting the above snippet after every '{}', '+' or '*' quantifiers and having that code cause reading more data and re-trying the match every time when triggered will satisfy both the incomplete match and the fail due to end-of-buffer problem.

Update...2008-02-28 08:10 CST: Noted the all important '?' quantifier was missing from the code snippet. Not now!

In reply to How regexes fail. by jdoege

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.