Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I have a sensor that sends data using an HTTP write that I want to log. I am new to socket programming but found an example in O’Reilly’s “Advanced Perl Programming”. After adjusting the code for my network, I get the input properly from the sensor and am able to process it without problems. The code is:
#!/usr/local/bin/perl # # usage: # use warnings; use strict; use IO::Socket; open PIDfile, '>', '/var/run/aws.rcvr.pid'; print PIDfile "$$\n"; close PIDfile; my ($sock, $new_sock, $buf ); $sock = new IO::Socket::INET (LocalHost => '192.168.100.7', LocalPort => 8081, Proto => 'tcp', Listen => 1, Reuse => 1 ); die "Socket could not be created. Reason: $!" unless $sock; while ($new_sock = $sock->accept()) { while (defined ($buf = <$new_sock>)) { if (index($buf, "/DATA_String/") > 0) { system("sh", "/home/app_scripts/process.input", "$buf"); # } else { } } } close ($sock);

I run the code on a FreeBSD 13.0 system in background. However I am doing something wrong (or have missed something) because I get the error:
 +sonewconn: pcb 0xfffff8003e9f4d90 (192.168.100.7:8081 (proto 6)): Listen queue overflow: 2 already in queue awaiting acceptance (6 occurrences)
repeatedly in the system logs. I tried increasing the buffers but that just increased the number in the queue. Can someone explain what is wrong or point me to a more appropriate example?
TIA - JJ

Replies are listed 'Best First'.
Re: Socket buffer errors
by haukex (Archbishop) on Apr 14, 2023 at 21:08 UTC

    I would strongly recommend against writing a HTTP server at such a low level. There are lots of ways to implement an HTTP server in Perl, and by now I've written quite a few that accept data from sensors and data loggers, I even gave a talk on that kind of thing. In the past I've implemented them using modules such as HTTP::Server::Simple (though I wouldn't really recommend that one anymore), CGI or CGI::Fast behind lighttpd (also somewhat outdated), Plack (modern but relatively low-level), and POE (fairly big learning curve), before finally settling on Mojolicious as my favorite. If you look at the bottom of my scratchpad, you'll see I've linked a whole bunch of Mojolicious example code I've posted here, and Mojolicious::Guides::Tutorial is a good place to start.

Re: Socket buffer errors
by kcott (Archbishop) on Apr 14, 2023 at 22:00 UTC

    There's not enough information for me to test your code; however, this condition strikes me as potentially problematic:

    index($buf, "/DATA_String/") > 0

    In "index STR,SUBSTR,POSITION", the POSITION is zero-based. Perhaps you want:

    index($buf, "/DATA_String/") > -1

    Consider this code:

    $ perl -E ' my @strings = qw{ no_data_string /DATA_String/ X/DATA_String/ /some/path/to/DATA_String/ }; say "-" x 40; for my $string (@strings) { say "\$string[$string]"; for my $pos (qw{0 -1}) { say "\$pos[$pos]"; if (index($string, "/DATA_String/") > $pos) { say "index() TRUE"; } else { say "index() FALSE"; } } say "-" x 40; } '

    Output:

    ---------------------------------------- $string[no_data_string] $pos[0] index() FALSE $pos[-1] index() FALSE ---------------------------------------- $string[/DATA_String/] $pos[0] index() FALSE $pos[-1] index() TRUE ---------------------------------------- $string[X/DATA_String/] $pos[0] index() TRUE $pos[-1] index() TRUE ---------------------------------------- $string[/some/path/to/DATA_String/] $pos[0] index() TRUE $pos[-1] index() TRUE ----------------------------------------
    "I am new to socket programming but found an example in O’Reilly’s “Advanced Perl Programming”. After adjusting the code for my network, ..."

    "Advanced Perl Programming" is an excellent book, I own a copy myself; however, it was published over a quarter of a century ago. Here's some suggestions for aligning your code with modern practices:

    • Check for I/O exceptions — I'd recommend the autodie pragma.
    • Use a lexical variable for the filehandle in the smallest scope possible (see open()).
    • Avoid "Indirect Object Syntax".

    So, your code might become something like this:

    ... use strict; use warnings; use autodie; use IO::Socket; { open my $pid_fh, '>', '/var/run/aws.rcvr.pid'; print $pid_fh "$$\n"; } my ($sock, $new_sock, $buf ); $sock = IO::Socket::INET::->new(...); ...

    — Ken

Re: Socket buffer errors
by NERDVANA (Priest) on Apr 15, 2023 at 05:44 UTC

    I hadn't heard of that error before, so I looked it up and it sounds like a BSD thing.

    Anyway, the problem is that you specified a listen queue size of 1, and somehow, that data device is sending packets faster than you receive and process them. The error will occur specifically when the device has sent one connection and you haven't accepted it yet and then receives another from the device. All you need to do is increase Listen => 1 to something bigger and you can buffer more incoming requests until you can get around to receiving them.

    But, the problem might be in the code that handles them, or in a variety of other scenarios like if the socket doesn't cleanly close and leaves your end hanging waiting for more data on a socket that the device is no longer writing to. In addition to raising your Listen queue size, you should observe the timing of when you receive the message and when you finish processing it, i.e. by printing the current time at the end of your inner while loop.

Re: Socket buffer errors
by Marshall (Canon) on Apr 15, 2023 at 16:39 UTC
    Evidently the sender makes a TCP connection, sends some data, and then terminates the socket. You should terminate at your end and wait for a new request. Move the close socket statement up into the loop and of course, it should be on $new_sock.
    while ($new_sock = $sock->accept()) { while (defined ($buf = <$new_sock>)) { if (index($buf, "/DATA_String/") > 0) { system("sh", "/home/app_scripts/process.input", "$buf"); } } close ($new_sock); ########################### }
    There are some low-probability failure modes with the above code. There are some things beyond your control that could cause the accept() to fail, so normally the loop condition would be:
    while (1) { $new_sock = $sock->accept() || next; ....
    There can be some issues with the way that you are reading the socket depending on what the sender is actually doing. You will be ok as long as the entire record is contained in a single MTU (transmission unit).

    Update: If requests are piling up because you cannot process them in actual receive time, then I would just append the data line to a file instead of trying to process it before the next request comes. Have a separate program reading that file and processing line by line. Of course, that file will grow since more is coming in than is being processed. Or do some way of processing these data lines en masse - launching a process per line is "expensive".

Re: Socket buffer errors
by Anonymous Monk on Apr 15, 2023 at 01:09 UTC
    haukex / kcott - Thanks for the suggestions. As additional explanation, this routine only needs to accept and process input. No response is sent back to the sensor and only one sensor provides input. So the simplest routine possibly is preferred.
    kcott - Each message received consists of 5 lines. All messages are identical except for the sensor data in the request line.
       A request line.
       Three header lines (host, connection, user-agent).
       A blank line.
    The request line format is:
    GET /DATA_String/{300 continuous characters of sensor data}HTTP/1.1
    A message is received approximately every minute. Processing by the called routine requires approximately 22-24 msec. Everything seems to work except the listen queue doesn’t seem to clear after the input is read to the buffer resulting in the listen queue overflow.

      Drip-feeding us bits of information isn't really appreciated. Please read "How do I post a question effectively?" and "Short, Self-Contained, Correct Example".

      As it stands, we don't know anything about the client side beyond "sends data using an HTTP write"; we don't know what "/home/app_scripts/process.input" does; and "PIDfile" is a complete mystery (you open, write, close, and never use again).

      Here's a scenario that works. It does the sort of things that you say you want to achieve. It doesn't generate any error or warning messages. Adapt to your needs.

      ken@titan ~/tmp/pm_11151665_io_socket $ ls -l total 4 -rwxr-xr-x 1 ken None 387 Apr 15 13:51 client.pl -rwxr-xr-x 1 ken None 50 Apr 15 11:57 process.sh -rwxr-xr-x 1 ken None 464 Apr 15 13:58 server.pl -rw-r--r-- 1 ken None 200 Apr 15 14:01 test.log

      client.pl:

      #!/usr/bin/env perl use strict; use warnings; use IO::Socket; my $client = IO::Socket::INET::->new( Proto => 'tcp', PeerAddr => 'localhost', PeerPort => 55555, ) || die "Can't open client socket: $IO::Socket::errstr"; for my $send_num (1 .. 5) { $client->print(<<"EOT"); GET /DATA_String/{send_num = $send_num}HTTP/1.1 host connection user-agent EOT sleep 5; }

      server.pl:

      #!/usr/bin/env perl use strict; use warnings; use IO::Socket; my $server = IO::Socket::INET::->new( Proto => 'tcp', LocalPort => 55555, Listen => SOMAXCONN, ReuseAddr => 1, ) || die "Can't open server socket: $IO::Socket::errstr"; my $client = $server->accept(); $client->autoflush; while (defined(my $read = <$client>)) { chomp $read; if (index($read, 'GET /DATA_String/') == 0) { system('sh', 'process.sh', $read); } }

      process.sh:

      #!/usr/bin/env sh echo -e "$*" >> test.log 2>&1

      Start server.pl then start client.pl. These will finish after ~25secs. Now check the log.

      $ cat test.log GET /DATA_String/{send_num = 1}HTTP/1.1 GET /DATA_String/{send_num = 2}HTTP/1.1 GET /DATA_String/{send_num = 3}HTTP/1.1 GET /DATA_String/{send_num = 4}HTTP/1.1 GET /DATA_String/{send_num = 5}HTTP/1.1

      See also: IO::Socket; IO::Socket::INET; and, perlipc. The perlipc page has a lot of content; if you don't want to read it all, concentrate on "TCP Clients with IO::Socket" and "TCP Servers with IO::Socket" (which should answer questions you might have about my code).

      — Ken