Re^3: A question of fork efficiency

Testing on my system, I'm not sure the Timeout is working properly for connects.

Try this version with explicit timeouts for attempts over $timeout seconds.

Also, notice the extreme difference in 'user' and 'sys' times :)

#!/usr/bin/perl

# http://perlmonks.org/?node_id=1135109
# http://perlmonks.org/?node_id=11103970

use strict;
use warnings;
use IO::Socket;
use IO::Select;
use Time::HiRes qw( time );

my @ips = map "192.168.1.$_", 1..20; # your IPs here
my $port = 22; # your port here
my $max = 1000; # somewhat smaller than max "open files"
my $timeout = 1;

my %handles;
my $sel = IO::Select->new;

while( @ips or $sel->count )
  {
  if( @ips and $sel->count < $max )
    {
    my $ip = shift @ips;
    my $fh = IO::Socket::INET->new(PeerHost => $ip, PeerPort => $port,
      Proto => 'tcp', Blocking => 0);
    $handles{$fh} = ["$ip:$port", $fh, time]; 
    $sel->add($fh);
    }
  elsif( @ips ? $sel->count >= $max : $sel->count )
    {
    my @connects;
    for my $fh ( @connects = $sel->can_write($timeout) )
      {
      print $fh->connected ? '   ' : 'not', " alive  $handles{$fh}[0]\
+n";
      $sel->remove($fh);
      delete $handles{$fh};
      }
    if( not @connects )
      {
      my $time = time - $timeout;
      for my $key ( keys %handles )
        {
        if( $handles{$key}[2] < $time )
          {
          print "not alive  $handles{$key}[0]\n";
          $sel->remove( $handles{$key}[1] );
          delete $handles{$key};
          }
        }
      }
    }
  }
[download]

There are several different ways to do the timeouts, I'm curious how this one works out. Others may be slightly faster on wall clock, but use much more CPU.

Comment on Re^3: A question of fork efficiency Download Code

Replies are listed 'Best First'.
Re^4: A question of fork efficiency by trippledubs (Deacon) on Aug 07, 2019 at 13:02 UTC
Nice! I/O Multiplexed has 4,000 system calls compared to 45,000 with fork measured by strace. Surprised that fork doesn't get smoked in wall clock time. Love the ternary in the if statement, educative as usual.	[reply]
Re^5: A question of fork efficiency by holli (Abbot) on Aug 07, 2019 at 13:57 UTC
This? `elsif( @ips ? $sel->count >= $max : $sel->count )` [download] What you call educative I call overly clever and in this case outright obstructive. This is the equivalent when you get rid of the ternary `elsif( ( @ips && $sel->count >= $max ) \|\| $sel->count )` [download] Which is clearer and not even longer. And once you see it like this and realize `@ips` is not used in within the following block, it becomes apparent you can safely omit that and the conditional becomes `elsif( $sel->count )` [download] One should keep conditionals as simple as possible. Similarly is this `my @connects; for my $fh ( @connects = $sel->can_write($timeout) )` [download] suboptimal. Apart from, again, obstructing the conditional, one should keep declaration and initialization as close to one another as possible. `my @connects = $sel->can_write($timeout); for my $fh ( @connects )` [download] This is better. holli You can lead your users to water, but alas, you cannot drown them.	[reply] [d/l] [select]
Re^6: A question of fork efficiency by synless (Acolyte) on Aug 07, 2019 at 14:43 UTC
Agreed. This is clearer. There are still other issues I have found as far as purposely testing against DNS names you know don't exist. Some of those go to the void and get no output returned.	[reply]
Re^4: A question of fork efficiency by synless (Acolyte) on Aug 07, 2019 at 13:23 UTC
This version is slightly faster and much better on CPU time. My version wrecks the CPU. The only thing I need to do here is tweak this version to run with the same output as mine. I'll post the portion that actually runs the commands soon. I think something like this may make this tool run much better. Here is your code running with 1000 as max on the 7k+ servers: `real 0m56.481s user 0m2.515s sys 0m2.165s` [download] Using my default fork value as max of 100: `real 1m26.502s user 0m2.537s sys 0m2.270s` [download]	[reply] [d/l] [select]
Re^5: A question of fork efficiency by tybalt89 (Monsignor) on Aug 07, 2019 at 15:40 UTC
It's not much faster on wall clock time because the problem itself is dominated by timeouts. I'd be curious to see the wall clock times when run on only 'alive' servers.	[reply]
Re^6: A question of fork efficiency by synless (Acolyte) on Aug 08, 2019 at 16:04 UTC
Ask and you shall receive. Output for your non blocking version: `Good count is 3037 Bad count is 0 Total count is 3037 real 0m24.790s user 0m1.093s sys 0m1.046s` [download] Forking version `Good count is 3037 Bad count is 0 Total count is 3037 real 0m15.358s user 0m10.870s sys 0m19.205s` [download] The forking version is faster up to a point but really puts the CPU through the ringer compared to what you came up with. Once you start hitting around 5k servers or so is when your version starts to shine.	[reply] [d/l] [select]
Re^4: A question of fork efficiency by synless (Acolyte) on Aug 07, 2019 at 14:49 UTC
I started testing this again and there are still some issues with it I'm trying to work out. Sometimes when a server does not exist in DNS I get no output back for that server name. For example if I run this against this array: `qw ( server1 server2 server3 server4 doesnotexist server18 )` [download] I'll get the following output: `alive server1 alive server2 alive server3 alive server4 not alive server18` [download] As you can see the server called 'doesnotexist' just doesn't return output at all.	[reply] [d/l] [select]
Re^5: A question of fork efficiency by tybalt89 (Monsignor) on Aug 07, 2019 at 15:20 UTC
Interesting. I cannot replicate this on my system. I added code for handling the case when the IO::Socket fails, it will now give a message. I don't see (yet) any way now to not get output for each server:port. #!/usr/bin/perl # http://perlmonks.org/?node_id=1135109 # http://perlmonks.org/?node_id=11103970 use strict; use warnings; use IO::Socket; use IO::Select; use Time::HiRes qw( time ); my @ips = map "192.168.1.$_", 1..20; # your IPs here push @ips, 'doesnotexist.com', 'doesnotexist'; # test for non existent my $port = 22; # your port here my $max = 1000; # somewhat smaller than max "open files" my $timeout = 1; my %handles; my $sel = IO::Select->new; while( @ips or $sel->count ) { if( @ips and $sel->count < $max ) { my $ip = shift @ips; if( my $fh = IO::Socket::INET->new(PeerHost => $ip, PeerPort => $p +ort, Proto => 'tcp', Blocking => 0)) { $handles{$fh} = ["$ip:$port", $fh, time]; $sel->add($fh); } else { print "not exist $ip:$port\n"; } } elsif( @ips ? $sel->count >= $max : $sel->count ) { my @connects; for my $fh ( @connects = $sel->can_write($timeout) ) { print $fh->connected ? ' ' : 'not', " alive $handles{$fh}[0]\ +n"; $sel->remove($fh); delete $handles{$fh}; } if( not @connects ) { my $time = time - $timeout; for my $key ( keys %handles ) { if( $handles{$key}[2] < $time ) { print "not alive $handles{$key}[0]\n"; $sel->remove( $handles{$key}[1] ); delete $handles{$key}; } } } } } [download]	[reply] [d/l]
Re^6: A question of fork efficiency by synless (Acolyte) on Aug 07, 2019 at 18:03 UTC
One issue I'm seeing is that any server that gets to this point of the code: `if( not @connects )` [download] Causes a 1 to 2 second delay. Looks like these are servers in DNS but not pingable.	[reply] [d/l]
Re^7: A question of fork efficiency by tybalt89 (Monsignor) on Aug 07, 2019 at 18:18 UTC
Re^6: A question of fork efficiency by synless (Acolyte) on Aug 07, 2019 at 15:58 UTC
This should work the way I want. I'll rewrite it a bit to replicate how the current function works. Thanks for your help :)	[reply]