Usage of FSA::Rules to parse log files

glasswalk3r has asked for the wisdom of the Perl Monks concerning the following question:

Greetings Monks,

Do you have any experience using FSA::Rules module? I'm trying to use a Finite State Machine to parse a log file and thus the usage of FSA::Rules.

The module has some difficulties to use (the classes interfaces are a bit weird to use and the documentation lacks a good example of usage) but my problem is related to the state change.

The parser initiates in a state that should try to "guess" which is the content being read. Since the log files (in normal conditions) should start with a "welcome message", the state machine goes from "reading the first line" to the "greetings" state and keep reading the lines (in the greetings state) until a command prompt is found. After that, the "greetings" state should changed to "command_submission" state, where the prompt and command will be parsed so it's possible to identify what kind of output will be found in the next line.

With this logic in mind, the "greetings" state should goes to "command_submission" and never execute again. Until that happens, the next lines should be processed by it.

The problem is, I'm not being able to apply this logic to FSA::Rules: I've being playing around with FSA::Rules::rules and the strict parameter during object creation, but the switch method always finishes with the message:

Cannot determine transition from state "greetings" at C:\temp\monitor/Siebel/Srvrmgr/ListParser.pm line 297

Looking at the code details of the module I saw that if the switch method does not find any rules method that returns true it will raise the exception. The thing is, "greetings" state has only one possibility to go to another state ("command_submission") and should keep processing until this state is reached.

I "solved" this putting a "default" state (always returns true) inside rules going to the greetings state itself. It keeps calling itself until the prompt is found. I don't think so this is the correct way to do that and, besides, I would like to use on_exit to produce the output of the greetings read and every time "greetings" goes to rules, on_exit is executed.

It's a bit complicated to explain. I hope the code below to helps to understand better the problem.

package Siebel::Srvrmgr::ListParser;
use Moose;
use FSA::Rules;

use lib 'c:/temp/monitor';
use Siebel::Srvrmgr::ListParser::Output;
use Data::Dumper;

has 'parsed_tree' => (
    is     => 'rw',
    isa    => 'ArrayRef',
    reader => 'get_parsed_tree',
    writer => '_set_parsed_tree'
);

#has 'default_prompt' =>
#  ( is => 'rw', isa => 'Str', reader => 'get_prompt', writer => 'set_
+prompt' );

has 'has_tree' => ( is => 'rw', isa => 'Bool', default => 0 );

has '_list_comp_format' =>
  ( is => 'ro', isa => 'Str', reader => 'get_list_comp_format' );

has 'prompt_regex' => (
    is      => 'rw',
    isa     => 'RegexpRef',
    reader  => 'get_prompt_regex',
    writer  => 'set_prompt_regex',
    default => sub { qr/^srvrmgr(\:\w+)?>(\s[\w\s]+)?/ }
);

has 'hello_regex' => (
    is      => 'rw',
    isa     => 'RegexpRef',
    reader  => 'get_hello_regex',
    writer  => 'set_hello_regex',
    default => sub {
qr/^Siebel\sEnterprise\sApplications\sSiebel\sServer\sManager\,\sVersi
+on\s\d+\.\d+\.\d+\s\[\d+\]\sLANG_INDEPENDENT\s?/;
    }
);

has '_buffer' => (
    is      => 'rw',
    isa     => 'ArrayRef',
    reader  => 'get_buffer',
    writer  => '_set_buffer',
    default => sub { return [] }
);

has 'command' => (
    is      => 'rw',
    isa     => 'Str',
    reader  => 'get_command',
    writer  => 'set_command',
    default => ''
);

sub set_buffer {

    my $self  = shift;
    my $value = shift;

    if ( defined($value) ) {

        my $buffer_ref = $self->get_buffer();

        push( @{$buffer_ref}, $value );

        $self->_set_buffer($buffer_ref);

    }

}

sub clean_buffer {

    my $self = shift;

    $self->_set_buffer( [] );

}

sub count_parsed {

    my $self = shift;

    return scalar( @{ $self->get_parsed_tree() } );

}

sub clean_parsed_tree {

    my $self = shift;

    $self->has_tree(0);

}

sub set_parsed_tree {

    my $self   = shift;
    my $output = shift;

    if ( $self->has_tree() ) {

        my $old_parsed_tree = $self->get_parsed_tree();
        push( @{$old_parsed_tree}, $output );
        $self->_set_parsed_tree($old_parsed_tree);

    }
    else {

        $self->_set_parsed_tree( [$output] );

    }

    $self->has_tree(1);

}

sub append_output {

    my $self      = shift;
    my $data_type = shift;
    my $data_ref  = shift;

    my $output = Siebel::Srvrmgr::ListParser::Output->new(
        { data_type => $data_type, data_parsed => $data_ref } );

    $self->set_parsed_tree($output);

}

sub parse {

    my $self = shift;

    # array ref
    my $data_ref = shift;

    die "data parameter must be an array reference\n"
      unless ( ref($data_ref) eq 'ARRAY' );

    my $fsa = FSA::Rules->new(

        #        { strict => 1 },    #params
        first_line => {
            do    => sub { print "Starting reading the data\n" },
            rules => [
                command_submission => sub {

                    my $state = shift;
                    my $line  = $state->notes('line');

                    $line =~ s/\r\n//g;

                    return $line =~
                      /$state->notes('parser')->get_prompt_regex()/;

                },
                greetings => sub {

                    my $state = shift;
                    my $line  = $state->notes('line');
                    $line =~ s/\r\n//g;

                    return $line =~ $state->notes('parser')->get_hello
+_regex();

                  }
            ],
            message => 'First line read'

        },
        greetings => {
            do => sub {

                my $state = shift;

                if ( defined( $state->notes('line') ) ) {

                    print "hello\n";

                    $state->notes('parser')
                      ->set_buffer( $state->notes('line') );

                }
            },

            on_exit => sub {
                my $state = shift;
                print Dumper( $state->notes('parser')->get_buffer() );
            },

 #on_exit => sub { print Dumper( $state->notes('parser')->parse_greeti
+ngs() ) },
            rules => [
                command_submission => sub {

                    my $state = shift;

                    return ( $state->notes('line') =~
                          $state->notes('parser')->get_prompt_regex() 
+);

                },

                greetings => sub {
                    return 1;
                  } # :TRICKY:05-07-2011:arfreitas: should check why i
+s necessary to do this
            ],
            message => 'prompt found'
        },
        list_comp => {
            do => sub {
                my $state = shift;

                $state->notes('parser')->set_buffer( $state->notes('li
+ne') );
            },
            rules => [
                command_submission => sub {

                    my $state = shift;

                    return $state->notes('line') =~
                      /$state->notes('parser')->get_prompt_regex()/;

                  }
            ],
            on_exit => sub { $self->parse_list_comp() },
            message => 'prompt found'
        },
        command_submission => {
            do => sub {

                my $state = shift;

                print "got it\n";
                print $state->notes('line'), "\n";

                my $cmd =
                  ( $state->notes('line') =~
                      $state->notes('parser')->get_prompt_regex() )[1]
+;

                $state->notes('parser')->set_command($cmd);
            },
            rules => [
                list_comp => sub {

                    my $state = shift;

                    if ( $state->notes('parser')->get_command() eq 'li
+st_comp' )
                    {

                        return 1;

                    }
                    else {

                        return 0;

                    }

                  }
            ],
            message => 'command submitted'
        }
    );

    my $state;

    foreach my $line ( @{$data_ref} ) {

        chomp($line);

        unless ( defined($state) ) {

            $state = $fsa->start();
            $state->notes( parser => $self );

        }

        $state->notes( line => $line );

        $fsa->switch();

    }

    $fsa->done();

}

# nothing seems to be interesting to do with the greetings text
sub parse_greetings {
    my $self = shift;

    my @own_data;

    foreach my $line ( @{ $self->get_buffer() } ) {

        chomp($line);
        next if $line eq '';
        push( @own_data, $line );

    }

    $self->clean_buffer();

    return \@own_data;

}

sub parse_list_comp {

    my $self = shift;

    my $data_ref = $self->get_buffer();

    my @parsed_lines;

    # removing the command itself
    shift( @{$data_ref} );

    my $header_line = shift( @{$data_ref} );

    my @columns = split( /\s{2,}/, $header_line );

    # removing the lines after the columns header
    shift( @{$data_ref} );

    # removing the three last lines Siebel 7.5.3
    for ( 1 .. 3 ) {

        pop( @{$data_ref} );

    }

    foreach my $line ( @{$data_ref} ) {

        chomp($line);
        next if $line eq '';

        my @fields_values = split( /\s{2,}/, $line );

        my %line;

        my $list_len = scalar(@columns);

        for ( my $i = 0 ; $i < $list_len ; $i++ ) {

            $line{ $columns[$i] } = $fields_values[$i];

        }

        push( @parsed_lines, \%line );

    }

    $self->clean_buffer();
    return \@parsed_lines;

}

no Moose;
__PACKAGE__->meta->make_immutable;
[download]

package Siebel::Srvrmgr::ListParser::Output;
use Moose;

has 'data_type'   => ( is => 'rw', isa => 'Str' );
has 'data_parsed' => ( is => 'rw', isa => 'ArrayRef' );

no Moose;
__PACKAGE__->meta->make_immutable;
[download]

Thanks for the help,

Alceu Rodrigues de Freitas Junior
---------------------------------
"You have enemies? Good. That means you've stood up for something, sometime in your life." - Sir Winston Churchill

Comment on Usage of FSA::Rules to parse log files Select or Download Code

Replies are listed 'Best First'.
Re: Usage of FSA::Rules to parse log files by jethro (Monsignor) on Jul 06, 2011 at 08:56 UTC
Looking at the code details of the module I saw that if the switch method does not find any rules method that returns true it will raise the exception. The thing is, "greetings" state has only one possibility to go to another state ("command_submission") and should keep processing until this state is reached. I know nothing about the module, but a FSM generally does not have an implied "if nothing matched, stay in the state" rule. If you want to keep a state until you find the trigger for a different state you have to explicitly make a rule that goes from 'greetings' to 'greetings'. Have you seen those pictures with circles and arrows usually used to depict a FSM? If a state is kept until something happens, there is an explicit arrow from the circle representing the state to itself. It is not implicit and often you don't want it to be.	[reply]
Re^2: Usage of FSA::Rules to parse log files by glasswalk3r (Friar) on Jul 13, 2011 at 01:22 UTC
Well, in this case I must say that the module is working fine. :-). Looks like the "problem" was my lacking of knowledge about Finite State Machines. Thanks for you help, Alceu Rodrigues de Freitas Junior --------------------------------- "You have enemies? Good. That means you've stood up for something, sometime in your life." - Sir Winston Churchill	[reply]