Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight

EOF problem with Dancer streaming proxy

by dsheroh (Monsignor)
on Jan 12, 2017 at 13:59 UTC ( #1179444=perlquestion: print w/replies, xml ) Need Help??

dsheroh has asked for the wisdom of the Perl Monks concerning the following question:

I have a system which needs to act as a proxy to another of our servers, mainly for legacy support reasons. With the help of Dancer as a proxy, I've managed to get it mostly working with the following code:
return send_file( \'ignored', streaming => 1, callbacks => { override => sub { eval { my $client_connection = shift; my $ua = LWP::UserAgent->new; my $client; my $status = $ua->get($real_url, ':content_cb' => sub { my ($data, $resp) = @_; unless ($client) { my $headers_in = $resp->headers; my %headers_out = ( 'Content-Disposition' => sprintf('inline; filename="%s"', $file->{fileName}), ); for (qw( Content-Type Content-Length Keep-Alive Last-Mod +ified )) { $headers_out{$_} = $headers_in->header($_) if $headers_in->header($_); } $client = $client_connection->([$resp->code, [%headers_o +ut]]); } $client->write($data); }); if ($status->is_error) { my $headers_in = $status->headers; my %headers_out; for (qw( Content-Type Content-Length Keep-Alive Last-Modif +ied )) { $headers_out{$_} = $headers_in->header($_) if $headers_in->header($_); } my $client = $client_connection->([$status->code, [%header +s_out]]); $client->write($status->error_as_HTML); } 1; } or warn "Proxy failure: $@"; return; }, }, );
I say "mostly working" because, while the files are streamed successfully and browsers will accept them without complaint, wget and curl are less forgiving. Both of these command-line programs issue errors after the file is (successfully) received:
$ curl -sS -o rcvd https://foo/bar.pdf curl: (18) transfer closed with outstanding read data remaining $ wget https://foo/bar.pdf <...> 2017-01-12 14:48:58 (6.10 MB/s) - Read error at byte 3316868 (Success. +).Retrying. <proceeds to loop endlessly>
The byte at which wget reports the read error is always the last byte of the file (i.e., equal to the file size), leading me to suspect that an EOF marker isn't being handled properly. Possibly also relevant is that, while the proxying code copies the Content-Length header from the original source, the original source does not provide that header, so Content-Length is not actually set.

Using curl/wget to download the file directly from the original source works perfectly with no error messages issued.

Does anyone have any insights as to what the cause of the problem might be?

Edit: On further investigation, the problem does not appear to be with Dancer itself. Testing with Dancer's internal mini-server (using bin/ does not exhibit this problem. It only shows up in the production environment, which has Apache <-> Starman <-> Dancer. So now to work out whether the issue is with one of the other components individually or with the interactions between them.

Replies are listed 'Best First'.
Re: EOF problem with Dancer streaming proxy
by Corion (Patriarch) on Jan 12, 2017 at 14:09 UTC

    I've patched Dancer locally to not add the Content-Length header for streamed resources that don't have a content length. If I remember correctly, Dancer forces some headers on every response even if they are not needed or even harmful.

    I'm not sure what the proper response for streamed, non-delimited content types like text/event-stream should be but maybe someone can divine that from the standards...

      Strange. I am using Dancer 1.3202, the same as the version of the source you linked to, but wget doesn't show a Content-Length header as being present:
      $ wget -S http://foo/bar.pdf --2017-01-12 15:43:26-- http://foo/bar.pdf Resolving foo Connecting to foo:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Thu, 12 Jan 2017 14:43:26 GMT Server: Apache/2.4.10 (Debian) Content-Type: application/pdf Content-Disposition: inline; filename="bar.pdf" Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Transfer-Encoding: chunked Length: unspecified [application/pdf]
Re: EOF problem with Dancer streaming proxy
by gsiems (Deacon) on Jan 12, 2017 at 16:47 UTC

    As the author of Dancer as a proxy... I never really did get it working correctly/to my satisfaction for all clients.

    For whatever it may be worth-- I've since migrated to Dancer2 and the proxy piece works much better (the code is simpler also).

      If you don't mind sharing, what were some of the (other) problems you ran into with Dancer 1? And what does the corresponding Dancer 2 code look like?

        Streaming in any form was the main problem that I had w/Dancer 1. While moving to Dancer 2 was partially influenced by that, it was also because Dancer 2 has nicer routing and other features.

        The current code looks like:

        sub proxy_request { my $self; $self = shift if ( ( _whoami() )[1] ne (caller)[1] ); my ( $proxy_route, $file_name ) = @_; unless ($proxy_route) { Tranquillus::Util->return_error('BAD_QUERY'); } my $cb = sub { my $respond = $Dancer2::Core::Route::RESPONDER; require LWP; my $ua = LWP::UserAgent->new; my $writer; my %m; $ua->get( $proxy_route, ':content_cb' => sub { my ( $data, $response, $protocol ) = @_; if ( not $writer ) { my $h = $response->headers; my @ary = ( 'Cache-Control', 'Content-Length', 'Content- +Type', 'Last-Modified', 'Content-Disposition' ); foreach my $key (@ary) { if ( $h->header($key) ) { $m{$key} = $h->header($key); } } unless ( exists $m{'Content-Length'} ) { # RFC 7230 # $m{'Transfer-Encoding'} = 'chunked'; } if ($file_name) { $m{'Content-Disposition'} ||= 'attachment; fil +ename="' . $file_name . '"'; } $writer = $respond->( [ $response->code, [%m] ] ); } # Ensure that we have a valid writer... if ($writer) { $writer->write($data); } }, ); # Cleanup. # Ensure that we have a valid writer... if ($writer) { if ( exists $m{'Transfer-Encoding'} ) { $writer->write(undef); $writer->write("\r\n"); } $writer->close; } }; my $response = Dancer2::Core::Response::Delayed->new( # error_cb => sub { $weak_self->logger_engine->log( + warning => @_ ) }, cb => $cb, request => $Dancer2::Core::Route::REQUEST, response => $Dancer2::Core::Route::RESPONSE, ); return $response; }

        ... and can be found at:

        Update: fixed the github link.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1179444]
Approved by marto
Front-paged by kcott
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (6)
As of 2022-01-26 17:31 GMT
Find Nodes?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:

    Results (69 votes). Check out past polls.