First, a personal thanks for taking the time to respond Amon. It's clear you have a strong understanding of Marpa and your response to this question and my previous questions have been exceedingly useful. I have been reading through the documentation but I am finding it difficult to find what I need as there is much to absorb.

You are correct that the 'read' call was in my code, it was missed in my copy/paste. I have edited the main post to reflect this change.

I had assumed that the behavior here was just something that I did not understand, not that Marpa was incorrect or faulty. Your response confirms this. So a couple of followup questions.

1) Given this information, it would seem to me that you should be able to differentiate between the parser exhausting the input as opposed to successfully parsing the entire grammar. Is there some way to determine this?

After a bit of thought, I'm assuming that determining what constitutes 'parsing the entire grammar' would be a difficult problem in and of itself considering the way a parse tree is created. Exhausting the input string is probably a reasonable indicator of a grammar having been parsed successfully.

2) How would you resolve the problem in example 2 assuming this was part of a more complex grammar? Considering you are locked into a greedy match behavior on a per token basis, this seems difficult to resolve.

It appears that it is possible to differentiate a failed match vs. an exhausted input string. If the recognizer specifically rejects the string as unable to match, it will throw an error. If the input string is exhausted, but has conformed to the grammar requirements up till that point, the recognizer instead pauses the parse. This allows for the opportunity to add to the input string and resume parsing.

However, if the full grammar has not been matched and the recognizer is paused, the 'value' method will result in an undef return. An undef return from the 'value' method means there was no parse result, or a failure.

This is pretty much exactly what Amon was telling me above but I didn't quite put things together until I understood that you could pause the parsing, add to your input, and resume parsing.

Short story is don't rely on an error from the recognizer to determine whether the grammar was matched. An error clearly indicates a failed match, but no error can mean 'input string was exhausted, but does not yet match fully' or 'grammar was matched successfully'. To differentiate these, ensure the value returned from the 'value' method is defined. Thanks to the Marpa author Jeffrey for the insight on this point https://groups.google.com/forum/#!topic/marpa-parser/fZzhxdBDbGk

After some further testing, I still do not see the value being updated in the case of the partial (but apparently successful) parse. I have added the :default rule you suggest above to two cases. Example 1 has been modified to match all tokens in the grammar( '12' and '3' ). It does return the expected 'name, value' structure. However, Example 2 retains the original string that only matches the first token ('12'). If the parse was successful, I would expect at least the matched tokens to appear in the value data, but it does not appear to exist. I have taken the time to read through the read and value documentation you linked above, but this information does not seem to explain the results I see here (or at least I'm not making sense of it).

use strict;
use warnings;

use Marpa::R2;
use Data::Dumper;

# Example 1
{
    my $g_str = <<'END_GRAMMAR';
:default    ::= action => [name, value]
:start      ::= data
data        ::= '12' '3'
END_GRAMMAR

    my $g_obj = Marpa::R2::Scanless::G->new({ source  => \$g_str });
    my $p_obj = Marpa::R2::Scanless::R->new({
        grammar => $g_obj,
        trace_values => 1,
        trace_terminals => 1
    });

    my $s = '123';
    $p_obj->read( \$s );

    print "EXAMPLE 1 VALUE:".Dumper($p_obj->value)."\n";
}

# Example 2
{
    my $g_str = <<'END_GRAMMAR';
:default    ::= action => [name, value]
:start      ::= data
data        ::= '12' '3'
END_GRAMMAR

    my $g_obj = Marpa::R2::Scanless::G->new({ source  => \$g_str });
    my $p_obj = Marpa::R2::Scanless::R->new({
        grammar => $g_obj,
        trace_values => 1,
        trace_terminals => 1
    });

    my $s = '12';
    $p_obj->read( \$s );

    print "EXAMPLE 2 VALUE:".Dumper($p_obj->value)."\n";
}
[download]

Output:

Setting trace_terminals option
Setting trace_values option
Lexer "L0" accepted lexeme L1c1-2: '12'; value="12"
Lexer "L0" accepted lexeme L1c3: '3'; value="3"
EXAMPLE 1 VALUE:$VAR1 = \[
            'data',
            '12',
            '3'
          ];

Setting trace_terminals option
Setting trace_values option
Lexer "L0" accepted lexeme L1c1-2: '12'; value="12"
EXAMPLE 2 VALUE:$VAR1 = undef;
[download]

In example 2, given there is now a :default rule for returning match data from a structural rule, and that the parse was successful, why is there no output from value?

My guess is that since the 'data' token requires both '12' and '3' to match, 'data' was not successfully matched. This means there is no returned value for the token, resulting in an empty value data structure. But this sounds an awful lot like a grammar parse failure that should result in a ->read failure.

EDIT: As it turns out, I am making the faulty assumption that because the read method did not throw an error, the parse was successful. This is not necessarily true. See my post above Re^3: Marpa -- Partial grammar match does not result in failure