http://qs1969.pair.com?node_id=1232116

chrestomanci has asked for the wisdom of the Perl Monks concerning the following question:

I have a strange problem with perl clients to web service at work.

The service serves large files to windows clients running on many windows versions. The perl client under windows downloads the files using LWP. For historical reasons there are lots of different Perl versions. The server is either Perl + Mason + Apache2 or Python Tornado without a web server, in both cases inside a Docker container.

I have discovered that on recent windows versions, if the client is running Perl 5.24.0 or older, then on some occasions data sent from the Python server is lost, while apparently identical data from the Perl + Mason server arrives correctly. It looks like the http request is sent correctly, the server sends the reply, but the client does not receive it. I suspect that there is some sort of buggy interaction between Perl and the windows TCP/IP stack, but I don’t really know.

If I upgrade Perl to 5.24.1, then the bug disappears. I have observed this with both the Strawberry and Active State distributions of Perl.

So I am trying to work out what changed in the windows networking between the 5.24.0 and 5.24.1 versions of Perl. Naturally I have read the relevant perldelta, but I can’t see any changes that look likely, I have also looked at the source code diff in git, but there are too many changes to read easily, and as far as I can tell, no windows specific changes. I can’t find a list of bugs closed in 5.24.1.

Any ideas what the significant change is? If you could point me to a bug that was closed in 5.24.1 then that would be great.

Replies are listed 'Best First'.
Re: Windows networking changes in perl 5.24.1
by localshop (Monk) on Apr 04, 2019 at 15:26 UTC

    I understand the temptation to interpret the behaviour changes from 5.24.0 to 5.24.1 as suggesting that this is where to problem lies but often it isn't that clear-cut and I'd be surprised if this turned out to be a Perl version bug although it could possibly be tied to subtle behaviour changes for ambiguous code - seeing some code snippets may help rule this out.

    I'd be more inclined to look at the dependent module version changes - try to update LWP and Crypt::SSL IO::Socket::SSL + Net::SSLeay etc to most recent available versions.

    • Are you using strict and warnings?
    • Can you isolate a failing case URL consistently or is this intermittent and not URL specific
    • What are the MIME types of the responses? - eg are these file downloads or just plain text/html?
    • Have you tested the URLs with other download approaches other than LWP (curl CLI or other Perl HTTP client? Possibly test with Mojo::UserAgent)
    • Is the LWP Client running inside a docker container? Assume not but If so - What is the container O/S?
    • Does the LWP Client process terminate after receiving a result - could it be terminating before processing the response or mishandling buffer flush?
    • Are the LWP requests going to HTTP or HTTPS and is this consistent ?
    • What versions of LWP etc are bundled with each Perl version? Perhaps also check whether the module path libraries are distinct and not shared.
    • Have you confirmed this behaviour on Windows machines with only 5.24.0 installed and seen the behaviour rectify on a clean 5.24.1?
      • I'd first try use IO::Handle;OUT->autoflush(1) to see what happens.

        May also be worth looking at the HTTP headers included both client and server-side as tightening the HTTP dialogue parameters may resolve.

        Some small blocks of code that show the main LWP calls and a few lines of relevant log files could be useful to help others diagnose also

      > ... try to update LWP and Crypt::SSL

      There is no Crypt::SSL. It might be that you are referring to Crypt::SSLeay which was long time ago used to provide the SSL support in LWP. But since more than 8 years ago this is now done with IO::Socket::SSL + Net::SSLeay instead.

      Hi @localshop, thanks for taking an interest.

      • I am using strict & warnings.
      • The failing URLs are consistent, though I can't figure out any sort of pattern in either the URLs or the content they ought to download.
      • I am not sure if the server sets a MIME type at all. If it does it will be some sort of catch all for binary files like application/octet-stream
      • I have tried with Internet Explorer, Chrome & Firefox. All work fine. I have not tried with any command line tools or other perl libraries.
      • The LWP client is running on Windows (Versions 7, 8, 10 & server 2008) as a Virtual Machine running on Linux KVM virtualiser. Hosted on Ubuntu 14.04. The server is in a Docker container under CentOS & Rancher 1.5x.
      • LWP requests are in the clear on HTTP port 80.
      • Not sure about LWP versions. Will that be built in to the Perl distro, or separately installed?
      • I have confirmed this behaviour by installing later versions of perl on top of each other until the code starts working.

      The overall application is an internal file storage system. The client is fairly simple, and is mostly concerned with supplying authentication credentials, checking the SHA1 checksum of the file as it is downloaded, and re-trying failed downloads. The server checks credentials, records the fact that the client downloaded the file in a database, and finds the file in one of a set of caches, local and remote file servers.

      I can't upload a code example at the moment because I am not at work, but I will create and upload a minimal sample when I get a chance.

        Here is a code fragement, scrubed of all the company internal stuff:

        sub getfile { my ( $url, $outfile, $options ) = @_; $options ||= {}; # Default option values my $max_retries = $options->{'max_retries'} || 5; my $file_host = $options->{'file_host'} || $ENV{FILE_HOST} || +"file-host.companyname.internal"; my $ua = $options->{'ua'} || LWP::UserAgent->new +( keep_alive => 1, cookie_jar => {} ); # Force to a positive integer. $max_retries = int($max_retries); $max_retries = 1 if $max_retries < 1; # Construct auth headers my %auth_headers = (); if( $options->{'basic_auth_creds'} ) { %auth_headers = ( 'Authorization' => "Basic ".encode_base64($o +ptions->{'basic_auth_creds'}) ); } my ( $csum_type, $csum ) = csum_to_lookup($url); if ( $csum_type eq 'sha1' ) { $url = "http://$file_host/getfile/" . $csum; $wanted_sha1 = $csum; } else { ... # Look in the database to convert to SHA1 } my $result; ATTEMPT: foreach my $attempt ( 1 .. $max_retries ) { debug("attempt $attempt to get url: '$url'"); my $f_out; # This will contain the sha1 of the downloaded data once the d +ownload is complete my $digest = Digest->new('SHA-1'); # NB: This uses the request method on LWP::UA, that takes an i +nstance of HTTP::Request and a callback function. # See: https://metacpan.org/pod/LWP::UserAgent#ua-request-requ +est-content_cb $result = $ua->request( GET($url, %auth_headers), # Func exported from HTTP::Re +quest::Common, returns an instance of HTTP::Request sub { my ( $data, $res ) = @_; if ( $res->is_success ) { unless ($f_out) { $f_out = open_outfile( $outfile, $nooverwrite +) or croak "unable to open $outfile\n"; } print $f_out $data; $digest->add($data) if defined $wanted_sha1; } }, ); unless( $result->request->uri->eq($url) ) { print "$url has redirected to ".$result->request->uri."\n" +; } # close file or flush the filehandle to disk # FIXME close or flush could fail (eg if insufficient disk spa +ce). this should be bubbled up if (ref($outfile)) { $outfile->flush(); } else { close $f_out if defined($f_out); } # For error messages if something goes wrong. my $resp_string = sprintf "%d - %s", $result->code, $result->m +essage; # Check the downloaded data is what we expect # (Unless we have no checksum when downloading an arbitrary UR +L) # TODO: this segment checks sha1 even if download fails - unne +cessary warning! if ( defined $wanted_sha1 ) { my $download_failed = 0; if( ! -f $outfile || 0 == -s $outfile ) { unlink $outfile unless ref($outfile); warn "http downloaded failed ($resp_string): Wanted SH +A1:$wanted_sha1 but got an empty file"; $download_failed = 1; } elsif( $digest->hexdigest ne $wanted_sha1 ) { unlink $outfile unless ref($outfile); warn "http downloaded failed ($resp_string): Wanted SH +A1:$wanted_sha1 but got:" . $digest->hexdigest; $download_failed = 1; } if( $download_failed ) { if( $attempt == $max_retries ) { my $download_url = $result->request->uri; my $hostname = $result->request->uri->host; my $timestamp = scalar gmtime(); if( my $packed_ip = gethostbyname( $hostname ) ) { my $file_server_ip = inet_ntoa($packed_ip); + # Using old style pure perl instead of a modern library warn "$timestamp : Request was to $download_ur +l on IP: $file_server_ip"; } else { warn "$timestamp : Request was to $download_ur +l but cannot resolve IP address for $hostname"; } } else { next ATTEMPT; } } } if ( $result->is_success ) { last ATTEMPT; } else { unlink $outfile unless ref($outfile); carp "$url could not be retrieved - $resp_string\n"; # Delay before the next attempt sleep 2**$attempt unless $attempt == $max_retries; } } return wantarray() ? ( $result->is_success(), $result ) : $result- +>is_success(); }
Re: Windows networking changes in perl 5.24.1 (Module versions)
by Anonymous Monk on Apr 03, 2019 at 21:26 UTC
    Module versions