redss has asked for the wisdom of the Perl Monks concerning the following question:

I have a CGI script that uses the LWP module to retrieve a file with a given url and prints it back out. 
$buffer = get($url);
print "Content-Type: $mimetype\n\n";
print $buffer;

[download]

Unfortunately it has to read the whole file in before printing it back out which takes too long for large files.

Is there a way to open an output pipe to make it streaming, so that it prints out the data as its recieved?

thanks in advance!

Replies are listed 'Best First'.
Re: how to pipe url back to stdout as stream
by ikegami (Patriarch) on Dec 03, 2004 at 17:44 UTC

    yes, I think you can do it using the alternative syntaxes for User::Agent's request.

    $request = HTTP::Request->new('GET', 'http://...');
$response = $ua->request($request, '/tmp/sss');

    [download]

    or

    $request = HTTP::Request->new('GET', 'http://...');
$response = $ua->request($request, \&callback);
sub callback {
   my ($data, $response, $protocol) = @_;
   ...
}

    [download]

    Update: Working code:

    Read more... (860 Bytes)

[reply]
[d/l]
[select]
      Thats exactly what I needed! Great job - thanks!!!
[reply]
      Ok here's where I am now... I was writing a hotlink bypass script (not to be abused) and I've modified your script to suit that purpose: 
      #!/usr/bin/perl
use strict;
use warnings;
use LWP ();

print "Content-type: text/html\n\n<html><body>no url given\n" unless l
+ength $ENV{QUERY_STRING};

my $ua = LWP::UserAgent->new();
my $request = HTTP::Request->new('GET', $ENV{QUERY_STRING} );
my $response = $ua->request($request, \&callback);

sub callback {
   my ($data, $response, $protocol) = @_;

   unless ($response->{'callback_first'}) {
      $response->{'callback_first'} = 1;
      my $content_type = join('; ', $response->content_type());
      print("Content-Type: $content_type\n");
      print("\n");
   }

   print($data);
}

      [download]

      This is probably nothing new to you, but anyways if put on a web server with the name "file", then visiting the url: 
      www.mywebserver.org/cgi-bin/file?http://www.geocities.com/myhome/me.jp
+g

      [download]
      will enable the geocities file to be hotlinked (for use off ebay etc)

      HOWEVER, what I see happening is that when I save the above file (which should be called me.jpg) it actually gets sent to me with the name file.jpg.
      Any idea on how to make the script preserve the original filename?

[reply]
[d/l]
[select]
        There's an HTTP header called "Content-Disposition" that lets you pass a filename back to the browser (among other things). Try Googling for info on correct usage.
[reply]
        Just an idea... You could locally save the image you're fetching, and simply to a redirect to the local copy the next time someone asks for it.
[reply]

Back to Seekers of Perl Wisdom