edimusrex has asked for the wisdom of the Perl Monks concerning the following question:

For some reason my logic isn't working. This is a nagios script I have written and the status output works as planned, but what I would like it to do is print out the list of domains and the days remaining until they expire. I had it coded differantly before without the fork but wanted to increase the performance. My issue now is the hash is empty upon completion. Seems like it has to do with the fork but not sure why. Here's what I got so far


#!/usr/bin/perl use warnings; use strict; use POSIX qw(strftime); use Time::Piece; use Cwd qw( abs_path ); use File::Basename; use File::Basename qw( dirname ); use LWP::Simple; use Parallel::ForkManager; use Date::Calc qw (Delta_Days); use Data::Dumper; my $pm = new Parallel::ForkManager(5); my $domains = dirname(abs_path($0)).'/domains.txt'; my %domainList; my $flag = 0; my $date = strftime "%Y-%m-%d", localtime; my %month = ( 'jan'=>'01','feb'=>'02','mar'=>'03','apr'=>'04','may'=>' +05','jun'=>'06','jul'=>'07','aug'=>'08','sep'=>'09','oct'=>'10','nov' +=>'11','dec'=>'12' ); open my $fh, $domains or die "CRIT: Unable to open $domains: $!\n"; while( my $domainName = <$fh> ) { chomp $domainName; print $domainName."\n"; $pm->start and next; my $expDate = `jwhois -n -h whois.crsnic.net $domainName | grep Ex +piration | awk '{print \$3}'`; print $expDate; my $diff = &dateDiff($expDate); if ($diff < 28 and $diff > 14) { $flag = 1; } elsif ($diff <= 14) { $flag = 2; } $domainList{$domainName} = $diff; exit($diff); $pm->finish; } close $fh; $pm->wait_all_children; print Dumper(\%domainList); if ($flag == 2){ my $status = "CRIT: There are Domains Expiring Soon. Please Resol +ve"; print "$status\n"; for my $key (sort(keys(%domainList))) { print "$domainList{$key}\t +--- $key\n"; } exit 2; } elsif ($flag == 1){ my $status = "WARN: There are Domains Expiring within a month. Pl +ease Resolve"; print "$status\n"; for my $key (sort(keys(%domainList))) { print "$domainList{$key}\t +--- $key\n"; } exit 1; } else { my $status = "OK: Domains Look good"; print "$status\n"; for my $key (sort(keys(%domainList))) { print "$domainList{$key}\t +--- $key\n"; } exit 0; } sub dateDiff { my $ex = shift; chomp $ex; my($day,$mon,$year) = split("-",$ex); my ($tyear,$tmon,$tday) = split("-",$date); print "$tyear, $tmon, $tday, $year, $month{$mon}, $day\n"; my $remaining_days = Delta_Days($tyear, $tmon, $tday, $year, $mont +h{$mon}, $day); return $remaining_days; }

As always any help is greatly appreciated

As you see I print out the hash just to test but it's always empty. I am sure I am missing something simple here

Replies are listed 'Best First'.
Re: Creating a hash within a fork
by choroba (Cardinal) on Jun 04, 2015 at 17:06 UTC
    Variables are not shared between forks. See forks::shared.
    لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
      Thanks for the reply, I will look into that.

        Not sure if this is a big help to you, but you can use the return code of the child processes to communicate with the parent process. I did that in this case:

        $pm->run_on_finish( sub { my ($pid, $exit_code, $ident) = @_; $connected_server_count += $exit_code; $progress = ($count / $total_server_count) * 100; printf "Progress: %4.1f\%\r",$progress; $count++; } );
        Example of a bad (0) return:
        $pm->finish(0); # Couldn't connect to this one, skip to next server in + main loop
        Example of a good (1) return:
        # Close ssh connection to this server $ssh->close(); $pm->finish(1);

        The point being that $count kept track of how many servers were processed ($connected_server_count is used elsewhere) but it based that count on the return code. In this code, a server returned 0 if it could not connect, but returned 1 if it could, so I just summed those values. I've not played around with it, but I assume you can map out any numeric code to suit your purposes, perhaps the number of days, etc.

Re: Creating a hash within a fork
by edimusrex (Monk) on Jun 04, 2015 at 17:05 UTC
    I just realized that since it is being forked, each process is independent so I am assuming the hash reference isn't being shared across all the processes. I am not sure how to get around that. Thanks again.
      I am not sure how to get around that.

      Sharing a hash between threads is easy. Not fast; but easy.

      Alternatively, if you're scared of threads; consider looking MCE which provides a mechanism for appearing to share data between forks that is pretty transparent to the using code.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
      In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

      I apologize, but MCE::Shared is not available on CPAN at this time. MCE::Shared supporting processes is coming soon. It will be released with MCE 1.7. But there is hope and will use MCE's gather capability in a subsequent post.

Re: Creating a hash within a fork
by marioroy (Prior) on Jun 05, 2015 at 03:00 UTC

    In summary, one can take serial code and enable parallelism with little code.

    open my $fh, $domains or die "CRIT: Unable to open $domains: $!\n"; my %domainList; while( my $domainName = <$fh> ) { chomp $domainName; my $expDate = `jwhois -n -h whois.crsnic.net $domainName | grep Ex +piration | awk '{print \$3}'`; my $diff = &dateDiff($expDate); if ($diff < 28 and $diff > 14) { $flag = 1; } elsif ($diff <= 14) { $flag = 2; } $domainList{$domainName} = $diff; } close $fh;

    Parallelism is possible simply by wrapping MCE around serial code. This does not fork per each input item. Thus, graceful to the OS.

    use MCE::Loop chunk_size => 1, max_workers => 5; my %domainList = mce_loop_f { my $domainName = $_; chomp $domainName; my $expDate = `jwhois -n -h whois.crsnic.net $domainName | grep Ex +piration | awk '{print \$3}'`; my $diff = &dateDiff($expDate); if ($diff < 28 and $diff > 14) { $flag = 1; } elsif ($diff <= 14) { $flag = 2; } MCE->gather($domainName, $diff); } $domains;

    MCE can also take a file handle as input. However, this is less efficient for large files due to involving the manager process.

    use MCE::Loop chunk_size => 1, max_workers => 5; open my $fh, $domains or die "CRIT: Unable to open $domains: $!\n"; my %domainList = mce_loop_f { my $domainName = $_; chomp $domainName; my $expDate = `jwhois -n -h whois.crsnic.net $domainName | grep Ex +piration | awk '{print \$3}'`; my $diff = &dateDiff($expDate); if ($diff < 28 and $diff > 14) { $flag = 1; } elsif ($diff <= 14) { $flag = 2; } MCE->gather($domainName, $diff); } $fh; close $fh;
Re: Creating a hash within a fork
by marioroy (Prior) on Jun 05, 2015 at 02:19 UTC

    Below, am showing a way using MCE::Loop and calling MCE->gather.

    #!/usr/bin/perl use warnings; use strict; use POSIX qw(strftime); use Time::Piece; use Cwd qw( abs_path ); use File::Basename; use File::Basename qw( dirname ); use LWP::Simple; use MCE::Loop chunk_size => 1, max_workers => 5; use Date::Calc qw (Delta_Days); use Data::Dumper; my $domains = dirname(abs_path($0)).'/domains.txt'; my $flag = 0; my $date = strftime "%Y-%m-%d", localtime; my %month = ( 'jan'=>'01','feb'=>'02','mar'=>'03','apr'=>'04','may'=>' +05','jun'=>'06','jul'=>'07','aug'=>'08','sep'=>'09','oct'=>'10','nov' +=>'11','dec'=>'12' ); my %domainList = mce_loop_f { # my ($mce, $chunk_ref, $chunk_id) = @_; # $_ is the same as $chunk_ref->[0] for chunk_size => 1 my $domainName = $_; chomp $domainName; print $domainName."\n"; my $expDate = `jwhois -n -h whois.crsnic.net $domainName | grep Ex +piration | awk '{print \$3}'`; print $expDate; my $diff = &dateDiff($expDate); if ($diff < 28 and $diff > 14) { $flag = 1; } elsif ($diff <= 14) { $flag = 2; } MCE->gather($domainName, $diff); } $domains; print Dumper(\%domainList); ...

    That enables parallelism and MCE->gather(...) to pass data back to the hash variable residing under the manager process. MCE::Loops (mce_loop_f) opens the $domains file. The 5 workers persist while running (think of a 5 worker pool). Thus, 5 forks, not more.

      The following does the same thing. Notice the use of MCE::Loop::init for specifying MCE options. One may pass more then 2 arguments to MCE->gather with this demonstration.

      #!/usr/bin/perl use warnings; use strict; use POSIX qw(strftime); use Time::Piece; use Cwd qw( abs_path ); use File::Basename; use File::Basename qw( dirname ); use LWP::Simple; use MCE::Loop; use Date::Calc qw (Delta_Days); use Data::Dumper; my $domains = dirname(abs_path($0)).'/domains.txt'; my $flag = 0; my $date = strftime "%Y-%m-%d", localtime; my %month = ( 'jan'=>'01','feb'=>'02','mar'=>'03','apr'=>'04','may'=>' +05','jun'=>'06','jul'=>'07','aug'=>'08','sep'=>'09','oct'=>'10','nov' +=>'11','dec'=>'12' ); my %domainList; MCE::Loop::init { chunk_size => 1, max_workers => 5, gather => sub { my ($domainName, $diff) = @_; $domainList{ $domainName } = $diff; } }; mce_loop_f { my $domainName = $_; chomp $domainName; print $domainName."\n"; my $expDate = `jwhois -n -h whois.crsnic.net $domainName | grep Ex +piration | awk '{print \$3}'`; print $expDate; my $diff = &dateDiff($expDate); if ($diff < 28 and $diff > 14) { $flag = 1; } elsif ($diff <= 14) { $flag = 2; } MCE->gather($domainName, $diff); } $domains; MCE::Loop::finish; print Dumper(\%domainList); ...

        I like this solution. It was super easy to implement into my code. Thank you so much

Re: Creating a hash within a fork
by jeffa (Bishop) on Jun 04, 2015 at 17:43 UTC

    I do not believe that forking is going to improve the performance of this script as it appears to be very much I/O bound. Each child that is forked is reading the same file from the beginning to the end, correct? What you need instead is to be able to split the file up into chunks and have each process work on unique chunks and return the time deltas. Do you really want to complicate the process just to save a potentially small amount of time?

    Oh i see now ... why don't you just output the work to STDOUT and use another script to collect the results? Seems much easier than trying to share data.

    jeffa

    L-LL-L--L-LL-L--L-LL-L--
    -R--R-RR-R--R-RR-R--R-RR
    B--B--B--B--B--B--B--B--
    H---H---H---H---H---H---
    (the triplet paradiddle with high-hat)
    

      Yea, you're right. For a simple check it doesn't seem worth it to share data between threads. I ended up omitting the multi threading and going with a more straight forward solutions

      #!/usr/bin/perl use warnings; use strict; use POSIX qw(strftime); use Time::Piece; use Cwd qw( abs_path ); use File::Basename; use File::Basename qw( dirname ); use LWP::Simple; use Date::Calc qw (Delta_Days); my $domains = dirname(abs_path($0)).'/domains.txt'; my %domainList; my $flag = 0; my $date = strftime "%Y-%m-%d", localtime; my %month = ( 'jan'=>'01','feb'=>'02','mar'=>'03','apr'=>'04','may'=>' +05','jun'=>'06','jul'=>'07','aug'=>'08','sep'=>'09','oct'=>'10','nov' +=>'11','dec'=>'12' ); open my $fh, $domains or die "CRIT: Unable to open $domains: $!\n"; while( my $domainName = <$fh> ) { chomp $domainName; my $expDate = `jwhois -n -h whois.crsnic.net $domainName | grep Ex +piration | awk '{print \$3}'`; my $diff = &dateDiff($expDate); if ($diff < 28 and $diff > 14) { $flag = 1; } elsif ($diff <= 14) { $flag = 2; } $domainList{$domainName} = $diff; } close $fh; if ($flag == 2){ my $status = "CRIT: There are Domains Expiring Soon. Please Resol +ve"; print "$status\n"; for my $key (sort(keys(%domainList))) { print "$domainList{$key}\t +--- $key\n"; } exit 2; } elsif ($flag == 1){ my $status = "WARN: There are Domains Expiring within a month. Pl +ease Resolve"; print "$status\n"; for my $key (sort(keys(%domainList))) { print "$domainList{$key}\t +--- $key\n"; } exit 1; } else { my $status = "OK: Domains Look good"; print "$status\n"; for my $key (sort(keys(%domainList))) { print "$domainList{$key}\t +--- $key\n"; } exit 0; } sub dateDiff { my $ex = shift; chomp $ex; my($day,$mon,$year) = split("-",$ex); my ($tyear,$tmon,$tday) = split("-",$date); my $remaining_days = Delta_Days($tyear, $tmon, $tday, $year, $mont +h{$mon}, $day); return $remaining_days; }

      Thanks