fireartist has asked for the wisdom of the Perl Monks concerning the following question:

I have browsed for modules but can't find anything that looks suitable, but then I don't have any idea where to start with this.

I wish to log all internet activity on a Windows 98 machine which connects to the internet via a modem.
It must support / allow ssl, and preferably any other connections that would normally go through a web-browser.
I wish to record the contents of <POST>'s.
I will probably be using regex's to decide on what to log (i.e. not bother logging image files, css files, and some others).

I guess this may involve setting the browser to use a proxy, but don't know what else is involved (ie, writing the proxy!).
It should preferably work with my ZoneAlarm firewall.

I'm not asking for anyone to write me a solution, just some friendly pointers to modules / docs that would help.
Thanks,
Carl

Replies are listed 'Best First'.
Re: logging internet activity on win32
by Dog and Pony (Priest) on Sep 03, 2002 at 17:02 UTC
    You can easily build proxies for your web browser with HTTP::Daemon and LWP::UserAgent, since they both uses HTTP::Request and HTTP::Response. Basically, you get the request from HTTP::Daemon, possibly examines it (Data:Dumper is very useful for details), get the page/whatever by using the request in LWP::UserAgent and return the resulting request (again, possibly after examining) to the browser via HTTP::Daemon.

    I wish I had some sample code here, but that is at work. Although it should be easy to figure out from the examples and the docs.

    Now, I haven't tried this with HTTPS, so I have no idea if it would work. It is also possible you may need to fire up separate proxies for different protocols too, I am not sure offhand and without anything to test with. :)

    ZoneAlarm should be no problem, you just allow perl to connect to the outside (it even asks for permission, like TPF, right?).

    If this doesn't cut it, you could go more low-level with IO::Socket::INET. Basically, you have open two sockets, one to communicate with the browser, and one to do the same with the internet. Beware though, that now you will need to grok HTTP to do this. The upside is that you can proxy just about anything, here is a small example of a IMAP proxy that does nothing but pass on the data. I intend to do something with this at some point, but for now it will serve as an example on what you can do - principle should be the same. Is probably lousy code though, is just proof of concept to myself... :)

    use IO::Socket; my $listener = IO::Socket::INET->new ( Listen => 5, LocalAddr => 'localhost', LocalPort => 143, Proto => 'tcp' ); while(defined(my $connection = $listener->accept)) { print "New connection\n"; my $sender = IO::Socket::INET->new("$host:143"); $connection->print($sender->getline()); ## open LOG, '>>imaplog.txt' or die "$!"; while(my $line = $connection->getline()) { my ($id) = $line =~ /^(\w+)/; ## print LOG "Q: $line"; $sender->print($line); my $total_answer = ''; while(my $answer = $sender->getline()) { ## print LOG "A: $answer"; $total_answer .= $answer; #$connection->print($answer); last if($answer =~ /^$id\s+(OK|NO|BAD)/i); } $connection->print($total_answer); } ## close LOG; $sender->close; $connection->close; undef $sender; undef $connection; }

    Maybe, just maybe any of this helps. :)


    You have moved into a dark place.
    It is pitch black. You are likely to be eaten by a grue.
Re: logging internet activity on win32
by Rex(Wrecks) (Curate) on Sep 03, 2002 at 16:51 UTC
    Hmm, define Internet Activity. There is a ton of stuff that you can log, to me it looks like you want to log content and sites, correct? Like a trap/monitor for a child/teen surfing porn?

    If that is the case, start looking into Net::Pcap to sniff and disassemble packets and parse from there. I'm not sure it works on Win32, but it should as it just uses libpcap which is available for Win32. I would also suggest that is you are looking for something in particular, start building a dictionary type data structure/file to run your packets against, if you find a match, save the entire transaction from buffer and parse further or look through it by hand.

    If you are just trying to graph trends in surfing, you will probably need a lot of resources, and you will also want to look into databases to store your raw data in, then run queries against the database to get statistics.

    "Nothing is sure but death and taxes" I say combine the two and its death to all taxes!
      Yes, I should have been more precise.
      I wish to log url's requested and POST data if any.
      That's all.
      I'll check out Dog and Pony's module suggestions below, and let you know how get on, Thanks.