Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I'm currently struggling with parsing the following output and creating key value pairs using ":" as the delimiter. The trouble is the output being on multiple lines. Here's my test code:
use strict; use warnings; use Data::Dumper; my %hash = (); my $string = ""; while(<DATA>) { $_ =~ s/\n/ /g; $string .= $_; } my @array = split (/\:\s+/, $string); print Dumper( @array ); __DATA__ VLAN : 1 Status : Enabled FID : 1 Name : DEFAULT VLAN VLAN Type: Permanent Last change: 2009-08-31 16:48:45 Egress Ports: host.0.1 Forbidden Egress Ports: ge.3.39 Untagged Ports: host.0.1
Any thoughts?

Replies are listed 'Best First'.
Re: Parsing issue
by vitoco (Hermit) on Sep 08, 2009 at 21:05 UTC

    Problems detected in this output:

    • It is a very special output format. You can recognize keys and values visually, but is hard to find a pattern for automation. You should probably have to write some code to parse each line in order, one at a time.
    • If field names (keys) are always the same, another approach should be to search for them and find the corresponding values from the full form.
    • Some values are in a different line than the field name as you said in your post.
    • Colon ":" alone cannot be used as a delimiter, because it appears in a timestamp. You already know that!
    • What is missing in your approach is a pattern to split multi columns lines, like the first one.
    • There are multiple words in some keys.

    The last point gave me an idea: Let all the words with exactly one space between them just before a colon be a key. So you have to be sure that a value need many spaces after it not to be considered as part of the key from the next line (trick one). Then, also split everywhere there are many spaces and treat them as another delimiter (trick two).

    #!perl use strict; use warnings; use Data::Dumper; my $string = ""; while(<DATA>) { $_ =~ s/\n/ /g; # (trick one) $string .= $_; } my %hash = split (/\s*:\s+|\s\s+/, $string); # (trick two) print Dumper( \%hash ); __DATA__ VLAN : 1 Status : Enabled FID : 1 Name : DEFAULT VLAN VLAN Type: Permanent Last change: 2009-08-31 16:48:45 Egress Ports: host.0.1 Forbidden Egress Ports: ge.3.39 Untagged Ports: host.0.1

    Output:

    $VAR1 = { 'Last change' => '2009-08-31 16:48:45', 'Status' => 'Enabled', 'Forbidden Egress Ports' => 'ge.3.39', 'FID' => '1', 'VLAN' => '1', 'Untagged Ports' => 'host.0.1', 'Egress Ports' => 'host.0.1', 'Name' => 'DEFAULT VLAN', 'VLAN Type' => 'Permanent' };

    You were too close, but now you have to be sure that all possible values of 'VLAN Type' should not be so long to left only one space before 'Last change' field.

      Thanks...there is also the issue of having two or more spaces in a value like 'Name' for instance could be 'Some VLAN with    spaces'

        Then, you can forget this trick and try some of the other ideas I gave, like parsing each row at a time and cut each record at fixed columns or use some other regexp to get field values.

        #!perl use strict; use warnings; use Data::Dumper; my %hash = (); my $string = ""; while(<DATA>) { if (/^(VLAN)\s+:\s(.*?)\s+(Status)\s+:\s(.*?)\s*$/) { ($hash{$1}, $hash{$3}) = ($2, $4); } elsif (/^(FID)\s+:\s(.*?)\s+(Name)\s+:\s(.*?)\s*$/) { ($hash{$1}, $hash{$3}) = ($2, $4); } elsif (/^(VLAN Type):\s(.*?)\s+(Last change):\s(.*?)\s*$/) { ($hash{$1}, $hash{$3}) = ($2, $4); } elsif (/^\s((Forbidden )?(Egress|Untagged) Ports):\s*$/) { $string = $1; } elsif ($string) { /^\s*(.*?)\s*$/; ($hash{$string}, $string) = ($1, ""); } } print Dumper( \%hash ); __DATA__ VLAN : 1 Status : Enabled FID : 1 Name : Some VLAN with spaces VLAN Type: Permanent Last change: 2009-08-31 16:48:45 Egress Ports: host.0.1 Forbidden Egress Ports: ge.3.39 Untagged Ports: host.0.1
        $VAR1 = { 'Last change' => '2009-08-31 16:48:45', 'Status' => 'Enabled', 'Forbidden Egress Ports' => 'ge.3.39', 'FID' => '1', 'VLAN' => '1', 'Untagged Ports' => 'host.0.1', 'Egress Ports' => 'host.0.1', 'Name' => 'Some VLAN with spaces', 'VLAN Type' => 'Permanent' };

        Here, I used "(.*?)\s*" to get a trimmed value of any type field, but you should change each of them to a specific pattern for dates, integers...

        Update: Forgot one field...

        I forgot to mention in my previous post that the if (pat1) {} elsif (pat2) {} elsif ... inside a while loop method is useful when data records is not always in the same order.

        In this case, where your "output" format is fixed, it's better (faster) to use a per line parsing method (no while):

        #!perl use strict; use warnings; use Data::Dumper; my %hash = (); $_ = <DATA>; /:\s(.*?)\s+\w+\s+:\s(.*?)\s*$/; $hash{'VLAN'} = $1; $hash{'STAT'} = $2; $_ = <DATA>; /:\s(.*?)\s+\w+\s+:\s(.*?)\s*$/; $hash{'FID'} = $1; $hash{'NAME'} = $2; $_ = <DATA>; /:\s(.*?)\s+\w+\s\w+:\s(.*?)\s*$/; $hash{'VTYPE'} = $1; $hash{'LASTM'} = $2; $_ = <DATA>; $_ = <DATA>; /^\s*(.*?)\s*$/; $hash{'EP'} = $1; $_ = <DATA>; $_ = <DATA>; /^\s*(.*?)\s*$/; $hash{'FEP'} = $1; $_ = <DATA>; $_ = <DATA>; /^\s*(.*?)\s*$/; $hash{'UP'} = $1; print Dumper( \%hash ); __DATA__ VLAN : 1 Status : Enabled FID : 1 Name : Some VLAN with spaces VLAN Type: Permanent Last change: 2009-08-31 16:48:45 Egress Ports: host.0.1 Forbidden Egress Ports: ge.3.39 Untagged Ports: host.0.1

        This way, you can control other things like key names:

        $VAR1 = { 'NAME' => 'Some VLAN with spaces', 'LASTM' => '2009-08-31 16:48:45', 'VTYPE' => 'Permanent', 'FID' => '1', 'VLAN' => '1', 'STAT' => 'Enabled', 'UP' => 'host.0.1', 'EP' => 'host.0.1', 'FEP' => 'ge.3.39' };

        Thinking a bit more on my first script, I realize that the string can be also modified in the following way: add a new delimiter just before what we detect as a field name (words separated by exactly one space, before any colon followed by a space), then split:

        #!perl use strict; use warnings; use Data::Dumper; my $string = ""; while(<DATA>) { $_ =~ s/\n/ /g; $string .= $_; } $string =~ s/((\w+\s)*\w+)\s*:\s+/\%$1%/g; $string =~ s/^\%(.*)/$1\%/; $string =~ s/\s+\%/\%/g; my %hash = split (/\%/, $string); print Dumper( \%hash ); __DATA__ VLAN : 1 Status : Enabled FID : 1 Name : Some VLAN with spaces VLAN Type: Permanent Last change: 2009-08-31 16:48:45 Egress Ports: host.0.1 Forbidden Egress Ports: ge.3.39 Untagged Ports: host.0.1
        $VAR1 = { 'Last change' => '2009-08-31 16:48:45', 'Status' => 'Enabled', 'Forbidden Egress Ports' => 'ge.3.39', 'FID' => '1', 'VLAN' => '1', 'Untagged Ports' => 'host.0.1', 'Egress Ports' => 'host.0.1', 'Name' => 'Some VLAN with spaces', 'VLAN Type' => 'Permanent' };

        I also changed the delimiter to another unused char, to differentiate it from the colon inside the time value when trimming out extra spaces.

        BTW, this was fun!

Re: Parsing issue
by toolic (Bishop) on Sep 08, 2009 at 20:09 UTC
    If you are trying to slurp the contents of DATA into a scalar variable, you could locally modify the $/ special variable:
    use strict; use warnings; use Data::Dumper; my $string; { $/ = ''; $string = <DATA> } $string =~ s/\n/ /g; my @array = split /:\s+/, $string; print Dumper( \@array ); __DATA__ VLAN : 1 Status : Enabled FID : 1 Name : DEFAULT VLAN VLAN Type: Permanent Last change: 2009-08-31 16:48:45 Egress Ports: host.0.1 Forbidden Egress Ports: ge.3.39 Untagged Ports: host.0.1

    Can you also show us what your hash should look like?