-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

I just love getting you to say it! Kinda like Pavlov's dog and ringing that bell. Of course I have to grant it to you, been there done that, sometimes done it twice! Might make a good slogan for stonehenge....

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

Well, uh, I'm just writing a module that's called HTTP::Proxy. ;-)

The goal is to be able to pass every request and response though a set of configurable Perl "filters".

Here's an example script that creates a anonymizing proxy, just like in merlyn's column (his script has been one of my inspirations, along with two others scritps by Abigail and Rafael Garcia-Suarez):

#!/usr/bin/perl -w
use HTTP::Proxy qw( :log );
use strict;

# a very simple anonymizing proxy
my $proxy = HTTP::Proxy->new;
$proxy->logmask( shift || NONE );    # log configuration

# the anonymising filter
$proxy->push_headers_filter(
    mime    => undef,    # apply this on any type of content
    request => sub {
        $_[0]->remove_header(qw( User-Agent From Referer Cookie ));
    },
    response => sub {
        $_[0]->remove_header(qw( Set-Cookie )),;
    },
);

$proxy->start;
[download]

(This code works with the development version.)

The proxy is not fully functional yet, though. One of my goal is to create a set of filters that would log the whole session, so that yet another script could (automatically) create a web robot from the session transcript.

If you are interested, there is a web site (older versions and CVS snapshot) and a mailing-list at http://http-proxy.mongueurs.net/. I am interested in help, and comments on the API (which is subject to change).

not Perl and a heavyweight (functionwise, not necessarily slow/bloated) is Squid Web Proxy Cache which you could probably make do whatever you can imagine.

#!/usr/bin/perl

use warnings;
use strict;

use POE qw(Component::Server::TCP Component::Client::HTTP
           Filter::HTTPD Filter::Stream);

POE::Component::Client::HTTP->spawn(Alias     => 'user_agent',
                                    Streaming => 1024);

POE::Component::Server::TCP->new
    ( Alias        => 'web_server',
      Port         => 8000,
      ClientFilter => 'POE::Filter::HTTPD',
      ClientInput  => \&client_input,
      InlineStates => { proxy_response => \&proxy_response }
    );

$poe_kernel->run();

sub client_input {
    my ($kernel, $heap, $request) = @_[KERNEL, HEAP, ARG0];

    # if request is actually response it just means that we've got an
    # error
    if($request->isa('HTTP::Response')) {
        $heap->{client}->put($request);
        $kernel->yield('shutdown');
        return;
    }

    $request->header(Connection => 'close');
    $kernel->post(user_agent => request => proxy_response => $request)
+;
}

sub proxy_response {
    my ($kernel, $heap, $request_packet, $response_packet)
        = @_[KERNEL, HEAP, ARG0, ARG1];

    my $request           = $request_packet->[0];
    my ($response, $data) = @$response_packet;

    unless($heap->{got_headers}) {
        $response->header(Connection => 'close');
        $heap->{client}->put($response);
        $heap->{got_headers}++;
    }

    if(defined $data) {
        $heap->{client}->set_output_filter(POE::Filter::Stream->new())
+;
        $heap->{client}->put($data);
    } else {
        $kernel->yield("shutdown");
    }
}
[download]

--
Ilya Martynov, ilya@iponweb.net
CTO IPonWEB (UK) Ltd
Quality Perl Programming and Unix Support UK managed @ offshore prices - http://www.iponweb.net
Personal website - http://martynov.org

HTH, --traveler