I thought I'd mention the solution (to the general problem) Dominus uses in his upcoming book 'Perl Advanced Techniques Handbook' - for appropriate values of solution, of course, the general caveats apply. For more explanation subscribe to his mailing list, you should then get access to the complete chapter, out of which the following code is taken: 

#!/usr/bin/perl
use strict;

open my $fh, $ARGV[0] or die "Couldn't open '$ARGV[0]': $!";

my $iter = records( blocks($fh), qr/\s*,\s*/ );

while ( defined(my $rec = $iter->()) ) {
    print $rec, "----\n";
}

sub blocks {
    my $fh = shift;
    my $blocksize = shift || 8192;
    sub {
        return unless read $fh, my($block), $blocksize;
        return $block;
    }
}

sub records {
    my $blocks = shift;
    my $terminator = @_ ? shift : quotemeta($/);
    my @records;
    my ($buf, $finished) = ("");
    sub {
        while (@records == 0 && ! $finished) {
            if (defined(my $block = $blocks->())) {
                $buf .= $block;
                my @newrecs = split /($terminator)/, $buf;
                while (@newrecs > 2) {
                    push @records, shift(@newrecs).shift(@newrecs);
                }
                $buf = join "", @newrecs;
            } else {
                @records = $buf;
                $finished = 1;
            }
        }
        return shift(@records);
    }
}

[download]

The blocks sub creates an iterator out of a filehandle which returns chunks of a certain blocksize from this filehandle. In this context an iterator is a reference to a function, which when called like this $iter->() returns the next item or undef when the sequence is exhausted. Other methods of creating such a block-creating iterator could be used instead of this sub.

The records sub creates an iterator that returns single records delimited by a terminator - much like the standard perl input iterator <$filehandle> in combination with $/. The difference is that a regular expression is used as a terminator, no longer a fixed length string.

-- Hofmator

In reply to Re: Regexes on Streams - Revisited! by Hofmator
in thread Regexes on Streams - Revisited! by tsee

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.