EOF problem with Dancer streaming proxy

dsheroh has asked for the wisdom of the Perl Monks concerning the following question:

I have a system which needs to act as a proxy to another of our servers, mainly for legacy support reasons. With the help of Dancer as a proxy, I've managed to get it mostly working with the following code:

  return send_file(
    \'ignored', streaming => 1, callbacks => {
      override => sub {
        eval {
          my $client_connection = shift;
          my $ua = LWP::UserAgent->new;
          my $client;

          my $status = $ua->get($real_url, ':content_cb' => sub {
            my ($data, $resp) = @_;

            unless ($client) {
              my $headers_in = $resp->headers;
              my %headers_out = (
                'Content-Disposition' =>
                  sprintf('inline; filename="%s"', $file->{fileName}),
              );
              for (qw( Content-Type Content-Length Keep-Alive Last-Mod
+ified )) {
                $headers_out{$_} = $headers_in->header($_)
                  if $headers_in->header($_);
              }
              $client = $client_connection->([$resp->code, [%headers_o
+ut]]);
            }

            $client->write($data);
          });

          if ($status->is_error) {
            my $headers_in = $status->headers;
            my %headers_out;
            for (qw( Content-Type Content-Length Keep-Alive Last-Modif
+ied )) {
              $headers_out{$_} = $headers_in->header($_)
                if $headers_in->header($_);
            }

            my $client = $client_connection->([$status->code, [%header
+s_out]]);
            $client->write($status->error_as_HTML);
          }
          1;
        } or warn "Proxy failure: $@";

        return;
      },
    },
  );
[download]

I say "mostly working" because, while the files are streamed successfully and browsers will accept them without complaint, wget and curl are less forgiving. Both of these command-line programs issue errors after the file is (successfully) received:

$ curl -sS -o rcvd https://foo/bar.pdf
curl: (18) transfer closed with outstanding read data remaining
$ wget https://foo/bar.pdf
<...>
2017-01-12 14:48:58 (6.10 MB/s) - Read error at byte 3316868 (Success.
+).Retrying.
<proceeds to loop endlessly>
[download]

The byte at which wget reports the read error is always the last byte of the file (i.e., equal to the file size), leading me to suspect that an EOF marker isn't being handled properly. Possibly also relevant is that, while the proxying code copies the Content-Length header from the original source, the original source does not provide that header, so Content-Length is not actually set.

Using curl/wget to download the file directly from the original source works perfectly with no error messages issued.

Does anyone have any insights as to what the cause of the problem might be?

Edit: On further investigation, the problem does not appear to be with Dancer itself. Testing with Dancer's internal mini-server (using bin/app.pl) does not exhibit this problem. It only shows up in the production environment, which has Apache <-> Starman <-> Dancer. So now to work out whether the issue is with one of the other components individually or with the interactions between them.

Comment on EOF problem with Dancer streaming proxy Select or Download Code

Replies are listed 'Best First'.
Re: EOF problem with Dancer streaming proxy by Corion (Patriarch) on Jan 12, 2017 at 14:09 UTC
I've patched Dancer locally to not add the `Content-Length` header for streamed resources that don't have a content length. If I remember correctly, Dancer forces some headers on every response even if they are not needed or even harmful. I'm not sure what the proper response for streamed, non-delimited content types like `text/event-stream` should be but maybe someone can divine that from the standards...	[reply] [d/l] [select]
Re^2: EOF problem with Dancer streaming proxy by dsheroh (Monsignor) on Jan 12, 2017 at 14:50 UTC
Strange. I am using Dancer 1.3202, the same as the version of the source you linked to, but `wget` doesn't show a Content-Length header as being present: `$ wget -S http://foo/bar.pdf --2017-01-12 15:43:26-- http://foo/bar.pdf Resolving foo Connecting to foo:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Thu, 12 Jan 2017 14:43:26 GMT Server: Apache/2.4.10 (Debian) Content-Type: application/pdf Content-Disposition: inline; filename="bar.pdf" Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Transfer-Encoding: chunked Length: unspecified [application/pdf]` [download]	[reply] [d/l] [select]
Re: EOF problem with Dancer streaming proxy by gsiems (Deacon) on Jan 12, 2017 at 16:47 UTC
As the author of Dancer as a proxy... I never really did get it working correctly/to my satisfaction for all clients. For whatever it may be worth-- I've since migrated to Dancer2 and the proxy piece works much better (the code is simpler also).	[reply]
Re^2: EOF problem with Dancer streaming proxy by dsheroh (Monsignor) on Jan 13, 2017 at 12:42 UTC
If you don't mind sharing, what were some of the (other) problems you ran into with Dancer 1? And what does the corresponding Dancer 2 code look like?	[reply]
Re^3: EOF problem with Dancer streaming proxy by gsiems (Deacon) on Jan 13, 2017 at 15:08 UTC
Streaming in any form was the main problem that I had w/Dancer 1. While moving to Dancer 2 was partially influenced by that, it was also because Dancer 2 has nicer routing and other features. The current code looks like: sub proxy_request { my $self; $self = shift if ( ( _whoami() )[1] ne (caller)[1] ); my ( $proxy_route, $file_name ) = @_; unless ($proxy_route) { Tranquillus::Util->return_error('BAD_QUERY'); } my $cb = sub { my $respond = $Dancer2::Core::Route::RESPONDER; require LWP; my $ua = LWP::UserAgent->new; my $writer; my %m; $ua->get( $proxy_route, ':content_cb' => sub { my ( $data, $response, $protocol ) = @_; if ( not $writer ) { my $h = $response->headers; my @ary = ( 'Cache-Control', 'Content-Length', 'Content- +Type', 'Last-Modified', 'Content-Disposition' ); foreach my $key (@ary) { if ( $h->header($key) ) { $m{$key} = $h->header($key); } } unless ( exists $m{'Content-Length'} ) { # RFC 7230 # http://tools.ietf.org/html/rfc7230#section-4 $m{'Transfer-Encoding'} = 'chunked'; } if ($file_name) { $m{'Content-Disposition'} \|\|= 'attachment; fil +ename="' . $file_name . '"'; } $writer = $respond->( [ $response->code, [%m] ] ); } # Ensure that we have a valid writer... if ($writer) { $writer->write($data); } }, ); # Cleanup. # Ensure that we have a valid writer... if ($writer) { if ( exists $m{'Transfer-Encoding'} ) { $writer->write(undef); $writer->write("\r\n"); } $writer->close; } }; my $response = Dancer2::Core::Response::Delayed->new( # error_cb => sub { $weak_self->logger_engine->log( + warning => @_ ) }, cb => $cb, request => $Dancer2::Core::Route::REQUEST, response => $Dancer2::Core::Route::RESPONSE, ); return $response; } [download] ... and can be found at: https://github.com/gsiems/tranquillus/blob/master/lib/Tranquillus/Proxy.pm Update: fixed the github link.	[reply] [d/l]