IO::Socket client does not detect when server network connection dies

bkchris has asked for the wisdom of the Perl Monks concerning the following question:

Fellow Monks-

I am a frequent visitor of the site but first time poster. I'm having an issue with my client not detecting when it no longer has a connection to the server it is connected to. The server is a network device, so I don't have any control of how that operates. The client needs to have a constant connection to the server, listen and parse any output. I am okay with that part; the disconnection is my issue...the client will hang and not reconnect, it must be restarted

I've created a simulation (see below), it is just your basic Perl server and client files. If the client cannot connect to the server at start-up, it will call recursive functions until it is able to create a connection. If the server script is stopped, the client script tries to reconnect until the server script is restarted. If the network connection from the server is removed, the client does not know about it and cannot reconnect without killing and restarting the process.

Is this normal? Is there a way to get around this? Thanks in advance!

server.pl

#!/usr/bin/perl
#server.pl

use strict;
use IO::Socket::INET;

$| = 1; #auto flush

my ($socket,$client, $peerAddress, $peerPort);

$socket = new IO::Socket::INET (
    LocalHost => '127.0.0.1',
    LocalPort => '5000',
    Proto => 'tcp',
    Listen => 5,
    Reuse => 1,
) || die "cannot create socket $!\n";

print "listening for client connection on 3083\n";

while(1) {
    # waiting for new client connection.
    $client = $socket->accept();

    $peerAddress = $client->peerhost();
    $peerPort = $client->peerport();

    print "Accepted New Client Connection From: $peerAddress $peerPort
+\n";

}

$socket->close();
[download]

client.pl

#!/usr/bin/perl
# client.pl

use strict;
use IO::Socket::INET;

# start connecting to socket
&connectToSocket(0);

exit;

# handles the socket retries and network device output
sub connectToSocket() {

    my $retry = shift;
    my $localt = localtime();
    
    print "retry\t$retry\n";
    $retry++;
    
    my $socket = &openSocket($retry);
    print "TCP Connection Success.\n";
    
    # process input from network device
    while (<$socket>) {
        print "$_";
    }
    
    # reconnect if socket drops
    &connectToSocket(0);
}

# creates the socket connection to the network device
sub openSocket() {

    my $retry = shift;
    my $socket = IO::Socket::INET->new (
        Proto            => "tcp",
        PeerAddr    => "192.168.1.105",
        PeerPort        => "5000",
        Timeout        => "1",
    ) || &connectToSocket($retry);
    
    $socket->autoflush(1);
    return $socket;
}
[download]

Comment on IO::Socket client does not detect when server network connection dies Select or Download Code

Replies are listed 'Best First'.
Re: IO::Socket client does not detect when server network connection dies by ikegami (Patriarch) on Jul 11, 2011 at 19:34 UTC
Is this normal? Yes. When the network is down, there's no way for the server to notify the client that it's closing the connection. Is there a way to get around this? Reconnect if you don't hear anything from the server after a certain period of time. See IO::Select. You could also use `alarm`. Optionally, also have the client periodically issue some command that has no effect other than causing the server to send something back.	[reply]
Re: IO::Socket client does not detect when server network connection dies by zentara (Cardinal) on Jul 11, 2011 at 19:15 UTC
See Knowing when a socket connection is dead and $Socket->connected Not Returning False? but you should be able to use the connected method of IO::Socket. See also blocking, non-blocking, and semi-blocking if you still are having problems. I'm not really a human, but I play one on earth. Old Perl Programmer Haiku ................... flash japh	[reply]
Re^2: IO::Socket client does not detect when server network connection dies by ikegami (Patriarch) on Jul 11, 2011 at 19:37 UTC
but you should be able to use the connected method of IO::Socket. No. See your second link.	[reply]
Re: IO::Socket client does not detect when server network connection dies by Marshall (Canon) on Jul 11, 2011 at 20:19 UTC
Is this normal? If you are strictly a consumer of data provided by the server, (you only do reads), there is no notification that the server died. You have to infer that it died by noticing that you haven't received anything from it for awhile, usually via an alarm timout via a SIGALRM signal. If you write to a socket where the reader has gone away, that triggers a SIGPIPE signal. Your recursive call is weird. You can chew a lot of stack space this way and for nought. Also if connect is failing due to something other than the timeout, your code could bombard the poor server with a huge number of requests and very quickly and all the while burning lots of stack. A simple while loop should suffice, like this or whatever you want anything except pushing deeper into the stack. `my $socket; while ( !$socket = IO::Socket::INET->new ( Proto => "tcp", PeerAddr => "192.168.1.105", PeerPort => "5000", Timeout => "1") ) { sleep 1; #or other backoff algorithm here }` [download]	[reply] [d/l]
Re^2: IO::Socket client does not detect when server network connection dies by halfcountplus (Hermit) on Jul 11, 2011 at 23:26 UTC
> Your recursive call is weird. You can chew a lot of stack space this way and for nought. Also if connect is failing due to something other than the timeout, your code could bombard the poor server with a huge number of requests and very quickly and all the while burning lots of stack. A simple while loop should suffice, like this or whatever you want anything except pushing deeper into the stack. Ditto. As ugly as it may seem, you need to institute a maximum number of times to retry (ie, not infinite recursion) and if that limit is reached, you are throwing a significant error and/or informing the CLIENT user. It also makes much sense to use a delay of some seconds in this loop, so you don't get bombed by the client. > See Knowing when a socket connection is dead The bit in the first reply by "castaway" is the tish: It's dead (disconnected), when can_read() indicates data available, but reading from it gives you no bytes. If you are paranoid like me you also give this a few chances in a loop, but regardless: that is the ONLY way to detect a dropped connection. Read() etc will not necessarily return an error.	[reply]
Re^3: IO::Socket client does not detect when server network connection dies by Marshall (Canon) on Jul 12, 2011 at 02:20 UTC
Sounds like you are agreeing with me. The code in the OP could get really nasty. There are things that will cause connect to fail that won't take the 1 second timeout time. Server could get flooded with millions of connect requests per second. I showed a fixed one second delay, but better is some variation to move us out of some possible "sync" with others trying to do the same thing. Many retry algorithms implement back-off at an exponential rate to some limit (does a few retries real fast, then starts lengthening the time between retries). The main point is not to make any recursive calls. `my $socket; my $max_tries = 50; my $cur_try = 0; while ( ($cur_try++ < $max_tries) and !$socket = IO::Socket::INET->new ( Proto => "tcp", PeerAddr => "192.168.1.105", PeerPort => "5000", Timeout => "1") ) { sleep 1; sleep 1 if (rand() > .5); #some variance is good }` [download] As far as doing something when the server goes away. A common scheme would be like this: `$SIG{ALRM} = 'ALRMhandler'; my $line; while ( alarm(30), $line=<$socket>, defined($line) ) { alarm(0); #turn off alarm ... process data in $line .... } ...here if $line is not defined.... ...another possible "server went away" condition... sub ALRMhandler { ...see discussion... }` [download] The read is blocking. We could get some characters, but not enough to get us to the \n character, we get "stuck" part way though the reading. So we set alarm to some number of seconds, I just randomly picked 30 seconds. When a defined string is returned, the statement completes and we move into the body of the loop. That immediately turns off the alarm and processes the data. Next iteration does the same thing. If for some reason, the read returns an undef value, then we are not in the loop and that means that server went away. What happens in the "fall through loop" or ALRM signal depends upon the application. Log an error message perhaps. Then maybe restart the app. redo label will go back to label:. I think this is "safe" in Perl. But heck even just completely restarting by exec'ing the same Perl script again might be just fine also. What's appropriate is up to the OP's goal with his application.	[reply] [d/l] [select]
Re: IO::Socket client does not detect when server network connection dies by locked_user sundialsvc4 (Abbot) on Jul 12, 2011 at 13:21 UTC
Logic like this is sometimes well designed using Finite State-Machine (FSM) logic ... for which there are, of course, Perl packages (if you can’t find a package that is even-more well suited to your purposes). I have found this approach to be very well-suited to ironing out the complexity from otherwise-ornery algorithms that need to “react to” unpredictably changing conditions.
Re^2: IO::Socket client does not detect when server network connection dies by bkchris (Initiate) on Jul 13, 2011 at 14:03 UTC
Thanks everyone for the feedback. I was to 'heart beat' the connection with IO::Select and can_read(). I'm sure I'll need to tweak it as I dig deeper, but this is a better start. `my $i = 1; while ( $i > 0 ){ if ($sel->can_read(20)) { my $line = <$socket>; print $line; } else { $socket->send(";"); } }` [download]	[reply] [d/l]