Hi Monks,
This problem is rather invovled in the nitty gritty of HTTP::Proxy. I am trying use HTTP::Proxy to proxy a large (slow) html file. The problem is that it seems to be trying to download the HTTP response before forwarding it on. At first I thought this was a property of the LWP::UserAgent, but the following code (from the cookbook) works fine: 

  use LWP::UserAgent;
  $ua = LWP::UserAgent->new;
  #$URL = 'http://whatever/file.html';

  my $expected_length;
  my $bytes_received = 0;
  my $req = HTTP::Request->new( GET => $URL );
  my $res =
     $ua->simple_request($req,
               sub {
                   my($chunk, $res) = @_;
                   $bytes_received += length($chunk);
                   unless (defined $expected_length) {
                      $expected_length = $res->content_length || 0;
                   }
                   if ($expected_length) {
                        printf STDERR "%d%% - ",
                                  100 * $bytes_received / $expected_le
+ngth;
                   }
                   print STDERR "$bytes_received bytes received\n";

               });
   print $res->status_line, "\n";

[download]

Gives the output:
0% - 32 bytes received
0% - 108 bytes received
0% - 232 bytes received
etc.

So given the user agent is capable of parsing the html in chunks like this, how can I get HTTP::Proxy to do it? The source code for HTTP/Proxy.pm (sub serve_connections) seems pretty similar to my example, so I can't work out why it isn't working :(

I'm only a perl dabbler, where should I be looking for help on this type of thing? Is here a good choice? Finally can I start hacking the HTTP::Proxy module? Can I drop in a replacement HTTP/Proxy.pm but use the system version for everything else? (given that I don't have root access on this machine).

Any help greatfully appriciated!

In reply to change HTTP::Proxy so it doesn't store-then-forward by Gaff

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.