nikita.af has asked for the wisdom of the Perl Monks concerning the following question:

Hi all, I'm looking for the best practice of processing columns in perl. My use case is to parse Unix iostat command output. The issue is that iostat output changes on different Linux releases however I want to make my script flexible to use it on all hosts. The issue: 1 - iostat output contains different number of columns but I need only some of them; 2 - Output may be different on some Linux releases. I think the proper way is to look into iostat header to find where is a column I need... Here is my code: 

# Usage:
#       ./zbx_linux_io.pl <no args specified> - returns a list of avai
+lable devices in JSON.
#       ./zbx_linux_io.pl $DEVICE_NAME OPTION - returns a I/O statisti
+cs of given device in JSON. OPTIONs: rps, wps, util.


use warnings;
use strict;

# Get io statistics
my @iostat_system = qx(iostat -x -p -k);

# Get all block devices and its partitions
my @lsblk_system = qx(lsblk -l);

# Array to store partitions
my @partitions;

# Hash to store Zabbix data
my %disk_data;

# Removing all except disk partitions
@lsblk_system = grep /part/, @lsblk_system;

# Removing empty lines from iostat output
@iostat_system = grep /\S/, @iostat_system;

# Filtering out all fields except partition name
foreach (@lsblk_system) {
# Converting spaces or tabs to comma
        s/\s+/,/g;
# Getting partition name
        my ($partition) = split (/,/, $_);
        push @partitions, $partition;
}

foreach my $partition (@partitions) {
        foreach my $io (@iostat_system) {
                $io =~ s/\s+/,/g;
                $io =~ s/p(\d)/$1/;
                if ($io =~ /$partition/) {
                        my ($device, $rrqms, $wrqms, $rs, $ws, $rkBs, 
+$wkBs, $avgrqsz, $avgqusz, $await, $rawait, $wawait, $svctm, $util) =
+ split (/,/, $io);
                        $disk_data{$device}{rps}        = $rs;
                        $disk_data{$device}{wps}        = $ws;
                        $disk_data{$device}{util}       = $util;
                }
        }
}

if (! @ARGV) {
        my $JSON =  "{ \"data\": [\n" . join(",\n", map { "\t".'{ "{#D
+EVICE_NAME}": "'.$_.'" }' } sort keys %disk_data) . "\n]}\n";
        print $JSON;
} elsif ((defined $ARGV[0]) && (defined $ARGV[1])) {
        print "$disk_data{$ARGV[0]}{$ARGV[1]}\n"
}

[download]
P.S.: Sorry for bad english.

Replies are listed 'Best First'.
Re: Proper way to work with columns
by Athanasius (Archbishop) on Dec 27, 2016 at 05:42 UTC
[reply]

      https://metacpan.org/source/MARKWKM/Test-Parser-1.9/lib/Test/Parser/Iostat.pm

      I thought it might be at least a model to steal code from, but it seems Test::Parser::Iostat uses a fixed set of columns 

        push @{$self->{data}}, {device => $i[0], rrqm => $i[1], wrqm => $i[2
+],
            r => $i[3], w => $i[4], rmb => $i[5], wmb => $i[6], avgrq 
+=> $i[7],
            avgqu => $i[8], await => $i[9], svctm => $i[10], util => $
+i[11],
            elapsed_time => $self->{elapsed_time}};

      [download]
      and he suggest he has seen the cols change on different releases.

[reply]
[d/l]

      Hello Athanasius, I want to avoid using CPAN modules as I don't have a root privileges to install modules. If it possible to parse the output by "native" perl code I prefer this way.

[reply]

        Hello Athanasius, I want to avoid using CPAN modules as I don't have a root privileges to install modules. If it possible to parse the output by "native" perl code I prefer this way.

        Hi,

        Root privileges are not required to install module see Yes, even you can use CPAN

[reply]
Re: Proper way to work with columns
by huck (Prior) on Dec 27, 2016 at 07:08 UTC

    Something like this maybe? 

    use strict; use warnings;

my @iostat_system = qx(iostat -x -p -k -d);

my @colnames;
my $ncols=0;
my %disk_data;
foreach my $io (@iostat_system) {
  chomp $io;
#  print $io."\n";
  my @cols=split(' ',$io);

  next unless ($io);
#  if ($cols[0]=~/^device/i) { # what if the device is named device?
   if ($ncols==0 && $cols[0]=~/^device/i) {
      @colnames=map {$_=~s/[^a-zA-Z0-9]//g;$_} @cols;
      $ncols=-1+scalar(@colnames);
#      print join(' ',@colnames)."\n";
      }
  elsif($ncols>0) {
     my $dev=$cols[0];
     for my $i (1..$ncols) { $disk_data{$dev}{$colnames[$i]}=$cols[$i]
+;}
     }
  }

#  use Storable; i dont need this !!!!
  use Data::Dumper;

  $Data::Dumper::Deepcopy=1;
  $Data::Dumper::Purity=1;
  $Data::Dumper::Sortkeys=1;
  $Data::Dumper::Indent=2;

print Dumper(%disk_data);

    [download]

    edit:dropped storable

    edit:fix case of device named device

[reply]
[d/l]

      Hi huch,

      Thanks, it looks like the way I was looking for. I think the line 29:

      for my $i (1..$ncols) { $disk_data{$dev{$colnames[$i]}=$cols[$i] +;}

      [download]

      should be:

      for my $i (1..$ncols) { $disk_data{$dev{$colnames[$i]}=$cols[$i] +$i;}

      [download]

      Am I correct?

[reply]
[d/l]
[select]

        Hi nikita.af, no, the line should be 

        for my $i (1..$ncols) { 
        $disk_data{$dev}{$colnames[$i]}=$cols[$i];
        }

        [download]
        If you use the download link you will get the real code. The lone + you reference is part of a continuation line indicator provided as a wraparound indicator by the <code>...</code>deliminator

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom