cmv has asked for the wisdom of the Perl Monks concerning the following question:

This was also posted to the POE mailing list...

Wise Monks-

In my environment there exists an application that creates a raw ascii data file on a given host in real time. I wanted to take this raw data and put it up in a GUI analysis window, so I wrote a POE-based Perl-Tk script, which uses POE::Wheel::FollowTail (Thanks Rocco, et. al.) as the data feed and uses Perl-Tk as the GUI. This works very well.

Now, the application has a "-binary" option, which will create a raw binary file on a given host in real time. I want my GUI analysis program to be able to work with this data as well. What should I do?

Being lazy, I was hoping I could just define a different filter for POE::Wheel::FollowTail, that instead of being line oriented, would be record oriented (each record begins with 0xfefefefe).

I currently have a binary parser that can parse the data. Assuming the binary data is slurped up into the scalar "$data", the following code works great:
Update: The regular expression shown here is very wrong (works in some basic cases, blows chunks on most fringe cases) - Please ignore it and see my next comment on this node for a much better example. 

while($data) {
    # Find next beginning-of-record flag & extract record...
    $data =~ s/^[^\xfe]*(\xfe\xfe\xfe\xfe[^\xfe]+)// || die "No luck";
    my $event=$1;

    # Send record to parser...
    $parser->parse( $event );

    # Print restult...
    print "events DUMP:\n", Dumper($parser->{events}), "\n"; }
}

[download]
Being very new to POE and filters, is there something already existing that I could use? I've toyed with modifying one of the following:

POE::Filter::RecordBlock
POE::Filter::Regexp
POE::Filter::Block

Do any of these make sense for what I want to do? Should I punt on using FollowTail and try something else?

Any advice is appreciated!

-Craig

Replies are listed 'Best First'.
POE::Wheel::FollowTail to read binary records - solved?
by cmv (Chaplain) on Mar 20, 2009 at 15:15 UTC
    Folks-

    Wow, no responses. Either this is a hard question, or nobody cares about doing something goofy like this :-)

    After playing around with things, I've decided to go with POE::Filter::Block because it provides a way to override the default encode and decode routines with ones of your own. The following code illustrates how using POE::Filter::Block in standalone mode, works fine for extracting variable length binary messages beginning with 0xfefefefe (however using it as a filter on POE::Wheel::FollowTail still doesn't work - more on this later).

    Update: Fixed buffering bug (removed '|$' in regular expression) 

    use strict;
use warnings;

my $data='ThisIsBeginningGarbagefefefefe037e01000001000a666c7832313100
+00000000000000003c0200000049b7e983000001030000015a000000000000000a003
+25b15000114b0fefefefe037e01000001000a666c78323131000000000000000036bc
+0200000149b7e983000001030000015b44988c1a0000000a00000028000114b002000
+00249b7e983000001030000015a00000000000000070032343200018c640200000349
+b7e983000001030000015b4475066c000000080000000400018c640200000449b7e98
+3000001030000015a0000000000000003003233da0001877d0200000549b7e9830000
+01030000015b44c0f03f00000006000000020001877d0200fe49b7e98300000210000
+0015c00000000014a00010000000000011475020000ff49b7e983000002200000015a
+000000000000000a0032a969000114d60200010049b7e983000002200000015b4427f
+d090000000300000005000114d60200010149b7e983000002200000015a0000000000
+0000110032a9010001132a0200010249b7e983000002200000015b4190ca460000000
+00000005c0001132a0200010349b7e983000002200000015a00000000000000110032
+f6ec000112410200010449b7e983000002200000015b411dadd4000000000000005c0
+00112410200010549b7e983000002200000015a00000000000000110032a90200fefe
+fefe037e01000001000a666c78323131000000000000000036bc0200000149b7e9830
+00001030000015b44988c1a0000000a00000028000114b00200000249b7e983000001
+030000015a00000000000000070032343200018c640200000349b7e98300000103000
+0015b4475066c000000080000000400018c640200000449b7e983000001030000015a
+0000000000000003003233da0001877d0200000549b7e983000001030000015b44c0f
+03f00000006000000020001877d0200000649b7e983000001030000015a0000000000
+0000070032340a00018b400200000749b7e983000001030000015b44407da20000000
+8000000040001';

$data = pack('H*', $data);

use POE::Filter::Block;

sub _myencode { print STDERR "_myencode\n" };
sub _mydecode {
  my $stuff = shift;
  print STDERR "M1: ", unpack('H72', $$stuff), "...\n";

  # Extract everything through 1st 0xfe{4} flag, up to the 2nd...
  $$stuff =~ /^(.*?\xfe{4}.*?)(\xfe{4})/s || return;

  # Return its measured length (in characters)...
  my $length = length($1);
  return($length);
}

my $filter = POE::Filter::Block->new( LengthCodec =>
    [ \&_myencode, \&_mydecode ]
);

$filter->get_one_start([ $data ]);

while (1) {
  my $block = $filter->get_one();
  last unless @$block;
  print STDERR "M2: ", unpack('H72', $block->[0]), "...\n",
    length($block->[0]), " characters\n";
}

    [download]
    My output looks like this (the first message includes the 'ThisIsBeginningGarbage' stuff by design) (truncated for eye appeal): 
    M1: d12c2cbe02772700abba0efefefefe037e01000001000a666c7832313100000000
M2: d12c2cbe02772700abba0efefefefe037e01000001000a666c7832313100000000
71 characters
M1: fefefefe037e01000001000a666c78323131000000000000000036bc0200000149
M2: fefefefe037e01000001000a666c78323131000000000000000036bc0200000149
440 characters
M1: fefefefe037e01000001000a666c78323131000000000000000036bc0200000149
M2: fefefefe037e01000001000a666c78323131000000000000000036bc0200000149
250 characters
M1: ...

    [download]
[reply]
[d/l]
[select]
Re: POE::Wheel::FollowTail to read binary records - solved!
by cmv (Chaplain) on Mar 20, 2009 at 16:26 UTC
    Folks-

    (however using it as a filter on POE::Wheel::FollowTail still doesn't work - more on this later).

    I take back this statement - everything seems to be working fine using the attached code. My apologies.

    Please feel free to comment on the code, or a better way to do this - it's my first time.

    Thanks

    -Craig

    Update: Fixed buffering bug (removed '|$' in regular expression) 

    use strict;
use warnings;

my $data='ThisIsBeginningGarbagefefefefe037e01000001000a666c7832313100
+00000000000000003c0200000049b7e983000001030000015a000000000000000a003
+25b15000114b0fefefefe037e01000001000a666c78323131000000000000000036bc
+0200000149b7e983000001030000015b44988c1a0000000a00000028000114b002000
+00249b7e983000001030000015a00000000000000070032343200018c640200000349
+b7e983000001030000015b4475066c000000080000000400018c640200000449b7e98
+3000001030000015a0000000000000003003233da0001877d0200000549b7e9830000
+01030000015b44c0f03f00000006000000020001877d0200fe49b7e98300000210000
+0015c00000000014a00010000000000011475020000ff49b7e983000002200000015a
+000000000000000a0032a969000114d60200010049b7e983000002200000015b4427f
+d090000000300000005000114d60200010149b7e983000002200000015a0000000000
+0000110032a9010001132a0200010249b7e983000002200000015b4190ca460000000
+00000005c0001132a0200010349b7e983000002200000015a00000000000000110032
+f6ec000112410200010449b7e983000002200000015b411dadd4000000000000005c0
+00112410200010549b7e983000002200000015a00000000000000110032a90200fefe
+fefe037e01000001000a666c78323131000000000000000036bc0200000149b7e9830
+00001030000015b44988c1a0000000a00000028000114b00200000249b7e983000001
+030000015a00000000000000070032343200018c640200000349b7e98300000103000
+0015b4475066c000000080000000400018c640200000449b7e983000001030000015a
+0000000000000003003233da0001877d0200000549b7e983000001030000015b44c0f
+03f00000006000000020001877d0200000649b7e983000001030000015a0000000000
+0000070032340a00018b400200000749b7e983000001030000015b44407da20000000
+8000000040001';

# Create the tmp file to tail later on...
$data = pack('H*', $data);
my $tmpfile = "tmpdata";
open(TMPDATA, ">$tmpfile") || die "Can't write file $tmpfile";
print TMPDATA $data;
close(TMPDATA);

use POE;
use POE::Wheel::FollowTail;
use POE::Filter::Block;

# Create custom length encoder/decoder...
sub _myencode { print STDERR "_myencode\n" };
sub _mydecode {
  my $stuff = shift;
  print STDERR "M1: ", unpack('H72', $$stuff), "...\n";
  $$stuff =~ /^(.*?\xfe{4}.*?)(\xfe{4})/s || return;

  my $length = length($1);
  return($length);
}

# Create a POE::Filter::Block with my custom length encoder/decoder...
my $filter=POE::Filter::Block->
  new(LengthCodec=>[\&_myencode,\&_mydecode]);

# Create POE::Wheel::FollowTail session using my customized filter...
POE::Session->create(
  inline_states => {
    _start => sub {
      $_[HEAP]{tailor} = POE::Wheel::FollowTail->new(
        Seek=>0,
        Filename => $tmpfile,
        Filter=>$filter,
        InputEvent=>"got_binary_message",
      );
    },
    got_binary_message => sub {
      print STDERR "M2: ", unpack('H72', $_[ARG0]), "...\n",
        length($_[ARG0]), " characters\n";
    },
  }
);

  POE::Kernel->run();
  exit;

    [download]
[reply]
[d/l]

Back to Seekers of Perl Wisdom