Hi Monks,
This problem is rather invovled in the nitty gritty of HTTP::Proxy. I am trying use HTTP::Proxy to proxy a large (slow) html file. The problem is that it seems to be trying to download the HTTP response before forwarding it on. At first I thought this was a property of the LWP::UserAgent, but the following code (from the cookbook) works fine:

  use LWP::UserAgent;
  $ua = LWP::UserAgent->new;
  #$URL = 'http://whatever/file.html';

  my $expected_length;
  my $bytes_received = 0;
  my $req = HTTP::Request->new( GET => $URL );
  my $res =
     $ua->simple_request($req,
               sub {
                   my($chunk, $res) = @_;
                   $bytes_received += length($chunk);
                   unless (defined $expected_length) {
                      $expected_length = $res->content_length || 0;
                   }
                   if ($expected_length) {
                        printf STDERR "%d%% - ",
                                  100 * $bytes_received / $expected_le
+ngth;
                   }
                   print STDERR "$bytes_received bytes received\n";

               });
   print $res->status_line, "\n";
[download]

Gives the output:
0% - 32 bytes received
0% - 108 bytes received
0% - 232 bytes received
etc.

So given the user agent is capable of parsing the html in chunks like this, how can I get HTTP::Proxy to do it? The source code for HTTP/Proxy.pm (sub serve_connections) seems pretty similar to my example, so I can't work out why it isn't working :(

I'm only a perl dabbler, where should I be looking for help on this type of thing? Is here a good choice? Finally can I start hacking the HTTP::Proxy module? Can I drop in a replacement HTTP/Proxy.pm but use the system version for everything else? (given that I don't have root access on this machine).

Any help greatfully appriciated!