Here's a quick non OO lexer that is very similar to the lexer Damian conway creates in his book Objected Oriented Perl (in the book, he blesses a subroutine that is almost exactly like this, and then adds some utility methods to go along with it.) This will break my $x = "my $x = \"my $x\""; into its components (keyword, variable, quoted string, and semicolon)
Note: all you have to do is define your tokens in terms of regular expressions. 


#!/usr/bin/perl -w

use strict;

my $tokens = {
                'KEYWORD' => 'my',
                'QSTRING' => '"(?:[^"]|\\")*"',
                'VAR'   => '\$\w+',
                'SEMI' => ';',
         };



open(FH,"$ARGV[0]") || die("Error opening $ARGV[0]: $!\n");
undef $/;
my $text = <FH>;
close(FH);

my $lexer = &create_lexer($tokens);
{
 my @token = &$lexer($text);
 last unless(defined($token[0]));
 print("Got: $token[0] => $token[1]\n") if(defined($$tokens{$token[0]}
+));
 redo;
}

sub create_lexer {
 my($tokens) = shift;
 my($sub,$code,$token,$key);
 foreach $key (keys(%$tokens)) {
  $code .= '$_[0] =~ s/\A\s*?(' . $$tokens{$key} . ')// ';
  $code .= "&& return('$key'," . '$1);' . "\n";
 }
 $code .= '$_[0] =~ s/\A\s*?(\S)// && return("",$1);';
 $code .= "return(undef,undef);\n";
 $sub = eval("sub { $code } ") || die($@);
 return($sub);
}

[download]


cephas

BTW: Did I mention that Damian Conway wrote a great book <bold>Object Oriented Perl</bold> (And just in case there was any doubt, I got my idea for this lexer from him, and not the other way around. :) )

In reply to Re: The Relation Between Lexers and Parsers by cephas
in thread The Relation Between Lexers and Parsers by beppu

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.