Also X-Posted to the HOP-Discuss mailing list.

OK, I confess I was lazy. I posted my query in hopes of someone supplying me with a quick answer and I went on and did other things. Given that I was Warnocked, I decided to dig into this.

What I think is the problem is how the lookfor() function was returning the last item of the stream. Running the following code gives us a clue:

  use Stream qw/node drop list_to_stream/;

  my @tokens = (
    node( OP  => '+' ),
    node( VAR => 'x' ),
    node( VAL =>  3  )
  );
  my $stream = list_to_stream(@tokens);

  use Data::Dumper::Simple;
  while (my $node = drop($stream)) {
    print Dumper($node, $stream);
  }
[download]

That prints out the following (tightened up a bit for clarity):

  $node   = [ 'OP', '+' ];
  $stream = [               # AoA
    [ 'VAR', 'x' ],
    [ 'VAL',  3  ]
  ];

  $node   = [ 'VAR', 'x' ]; 
  $stream = [ 'VAL',  3  ]; # not AoA

  $node   = 'VAL';
  $stream = 3;              # scalar
[download]

That doesn't look like what the parser is expecting. Prior to the tail, it was always returning an AoA. However, when it got to the tail, it merely returned an array reference. I fixed this by adjusting the parser that lookfor() returns.

The parser looked like this:

  my $parser = parser {
    my $input = shift;
    return unless defined $input;
    my $next = head($input);
    for my $i (0 .. $#$wanted) {
      next unless defined $wanted->[$i];
      return unless $wanted->[$i] eq $next->[$i];
    }
    my $wanted_value = $value->($next, $u);
    return ($wanted_value, tail($input));
  };
[download]

By testing the tail to see if it matches expectations, I get this:

  my $parser = parser {
    my $input = shift;
    return unless defined $input;
    my $next = head($input);
    for my $i (0 .. $#$wanted) {
      next unless defined $wanted->[$i];
      return unless $wanted->[$i] eq $next->[$i];
    }
    my $wanted_value = $value->($next, $u);
    my $tail = tail($input);

    # is this the last token from the stream?
    if ($tail && 'ARRAY' eq ref $tail && 'ARRAY' ne ref $tail->[0]) {
        $tail = [ $tail ];
    }
    return ($wanted_value, $tail);
  };
[download]

I also have to adjust my first test of the remaining stream:

  my $parser = concatenate(
    lookfor('OP'),
    lookfor('VAR'),
  );
  my ( $parsed, $remainder ) = $parser->($stream);
  is_deeply $parsed, [qw/+ x/],
    'concatenate should return the parsed values';
  is_deeply $remainder, [[ VAL => 3 ]], # AoA, not just an aref
    '... and the rest of the stream';
[download]

This allows the final concatenate to match the entire stream:

  $parser = concatenate(
    lookfor('OP'),
    lookfor('VAR'),
    lookfor('VAL'),
  );
[download]

Have I analyzed this incorrectly or did I set my test up incorrectly with the list_to_stream() function? Should drop() be modified instead?

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.