In fact I think I wrote a similar parser for a SoPW the other, see Design hints for a file processor. Since some of your tokens stretch multiple files you need a slightly more intelligent tokenizer though.

First of all thanks for the advice!

This looks like a classical application of a recursive descending parser that I introduced in RFC: Parsing with perl - Regexes and beyond.

I have difficulty understanding the article. In my humble opinion it is rather theoretical. Maybe I am not worthy ;-)

In fact I think I wrote a similar parser for a SoPW the other, see Design hints for a file processor. Since some of your tokens stretch multiple files you need a slightly more intelligent tokenizer though.

The "slightly" is an understatement. Some of my nightmare examples to illustrate the point:

In general: KEYWORD = value with value being of a certain type. The comments should be ignored and are not part of the value.

/* A 2-dimensional sequence as the value is being called in ODL */
KEYWORD = ((1,2) (3,4) (5,8) /* some comment */
            9,11))
[download]

/* A set as the value is being called in ODL */
KEYWORD = { RED, BLUE, /* some comment */
            GREEN, HAZEL }
[download]

/* A text string spanning multiple lines */
KEYWORD = "some text /* not a comment but part of the value! */
          more text
          even more text" /* this is again a comment*/
[download]

I have difficulty understanding the article. In my humble opinion it is rather theoretical. Maybe I am not worthy ;-)

No, it probably means that you are not the intended target audience. Or that I did a bad job at writing.

The "slightly" is an understatement. Some of my nightmare examples to illustrate the point:

You're right, I underestimated the complexity. I thought you could just take lines, and multiple lines if they contained non-closed quoted strings.

Still you should not give up hope. I wrote a simple lexer that works for the example you gave:

use strict;
use warnings;
use Data::Dumper;
use Math::Expression::Evaluator::Lexer qw(lex);

my $d = do { local $/; <DATA> };

my @tokens = (
    ['Commment',    qr{/\*.*?\*/}s, sub { return }],
    ['Identifier',  qr{[a-zA-Z_]\w+}],
    ['Number',      qr{\d+}],
    ['Operator',    qr{[=(),+-/*{}]}],
    ['Quoted String', qr{"[^"]*"}],
    ['Newline',     qr{\n}],
    ['Whitespace',  qr{\s+}, sub { return }],
);

print Dumper lex($d, \@tokens);

__DATA__
/* A 2-dimensional sequence as the value is being called in ODL */
KEYWORD = ((1,2) (3,4) (5,8) /* some comment */
            9,11))

/* A set as the value is being called in ODL */
KEYWORD = { RED, BLUE, /* some comment */
            GREEN, HAZEL }

/* A text string spanning multiple lines */
KEYWORD = "some text /* not a comment but part of the value! */
          more text
          even more text" /* this is again a comment*/
[download]

This is far from ideal, but it does tokenize the data in a meaningful way, and strips comments, but not those inside quoted strings.

(The lexer in Math::Expression::Evaluator::Lexer is quite simple and not iterator-like. If you don't want to read all input at once, you need to come up with something more sophisticated.)

For example you would define a table object:

tableobject : tablestart tabledef(s) tableend { tablefunc($item[1]) }

tablestart : 'OBJECT' '=' 'TABLE' "\n" { tablestartfunc() }

tabledef : columnnumber 
          | name 
          | bytes 
          | startbyte 
          | columnobject { columnstartfunc() }
          | rows 

columnnumber : 'COLUMN_NUMBER' '=' number /\n/ 
  { $return= recordcolumn($item{number});  }

number : /\d+/
[download]

UPDATE:
Whether you record the table information with the callbacks or by using the $return mechanism of Parse::RecDescent is your decision. In the former case tablefunc doesn't need the $item[1] parameter and columnnumber() doesn't need to set $return. In the latter case columnnumber would need a more elaborate return value, for example $return= ['columnnumber',$item{number}];

Thanks for the example/explanation. I will spend some time on Parse::RecDescent to check if I can really use it.

The example I gave was a bit simple, there are many different types of OBJECTS and the values can be a bit tricky (see my response to the post of moritz above.

That makes P::RD even better suited. The great advantage of P::RD is that you can and recursively (top-down or bottom-up) specify what is expected. So in your example you just define that keywordata is either a twodimsequence, a set or a text:

keywordata: twodimsequence | set | text
[download]

How these are defined in detail you can work out later without the complexity of thinking about where they are used. Some time later you might define that a twodimsequence is a list of numberpairs surrounded by brackets:

twodimsequence: '(' numberpair(s) ')'
[download]

You see, the definition of the language to parse is really easy and straightforward. The difficult part is mostly what to do with the parsed data. If you already know what your callback routines have to do, the only problems left are in the details.

One detail is how to cope with the comments inside your data files.

One idea (that might also help you with moritz and pc88mxer solutions) is to do the parsing in two phases. In the first phase you would delete all comments and fold data spread over multiple lines into one line, in the second phase you would to the parsing. Note that you can let more than one parser run over some text with P::RD, so it is well suited for that decoupling of problems

The other possibility is to define the SKIP expression in P::RD to be sometimes not only spaces but comments and newlines as well.

Or you just put 'comment(?)' or 'comment(s?)' in all places that a comment can be and define what a comment looks like (obviously no callback needed there). That may clutter your definitions but may be less tricky to get right than the SKIP method. The definition of a comment would probably look like this:

comment: m{/\* ([^*]|\*[^/])* \*/ }x
[download]

I don't say it is a walk in the park to get that parser running, the problems will be in the details. But you will be able to split the big problem into managable parts.

my $root = {};
my @stack;
my $object = $root;
use Data::Dumper;

while (<DATA>) {
  if (m{^\s*\/\*}) { next } # assume single line comment
  elsif (m/^\s*END(_OBJECT)?/) {
    $object = pop(@stack);
  } elsif (m/^\s*OBJECT\s*=\s*(\S+)/) {
    my $new = { parent => $object, type = $1 };
    push(@{$object->{children}}, $new);
    push(@stack, $object);
    $object = $new;
  } elsif (m/^\s*(\w+)\s*=\s*(\S*)/) {
    my $property = $1;
    my $value = $2;
    if ($value =~ m/^"/) {
      while ( (($value =~ tr/"//) % 2 != 0) && defined($_ = <DATA>)) {
        s/^\s*//;
        $value .= $_;
      }
    }
    $object->{$property} = $value;
  }
}

print Dumper($root);
[download]

Thanks for the effort!

You might consider writing a parser which is just "good enough" for the files you want to process. Since structures seem to end on line boundaries, you can try something like this: ...

Writing something just "good enough" was exactly my first approach:-) But now I want to improve and make it more generic. As you can read in my reply to moritz the values can be a bit tricky.