larsen has asked for the wisdom of the Perl Monks concerning the following question:

A good way to parse Apache logs I'd like to find a feasible way to parse Apache logs. I faced this problem some time ago, so I installed Apache::Parselog, but, as it has been mentioned in this thread (Apache::ParseLog), I encountered some problem.

Now I ask you what's your preferred way to parse and obtain informations from logs. Particularly, how do you deal with these problems?

Replies are listed 'Best First'.
Re: A good way to parse Apache logs
by echo (Pilgrim) on Aug 10, 2001 at 18:42 UTC
    Since you mention Apache::LogDBI I suspect you're interested in more than just parsing. Here's a solution that's scalable: use mod_log_spread. Each server in your web farm sends its log to a multicast group. You can then setup any number of listeners that will receive the logs and can act on them. This way you decouple not only parsing, but also storing the logs, from the web servers.
[reply]
Re: A good way to parse Apache logs
by stefan k (Curate) on Aug 10, 2001 at 18:13 UTC
    Hmm,
    this might not be what you want to hear, but anyway...
    Do you really need to write YAALP (yet another apache logfile parser)? I think there must be mere millions of them out there (maybe even more than email- and irc-clients *grin*)

    Of course this might be kind of a practice for you or you might need some information that those can't deliver. If so, please discard this posting to /dev/null :)

    Regards... Stefan

[reply]
[d/l]
[select]
      No, I think I'm not going to write a new logfile parser. Better, I'm not going to write a public available logfile parser. I think I'll write one just to learn why other parsers are better :)). What I'd like to find is the common used solution to perform this task (with solution I mean some composition of scripts and software, i.e. Perl + MySQL + something else...). And I'd like to know rationales that make this solution preferable.
[reply]
Re: A good way to parse Apache logs (HTTPD::Log::Filter)
by ChemBoy (Priest) on Oct 26, 2004 at 19:16 UTC

    This is just for the historical record, since I'm sure you solved your problem years ago, but I'm having some luck (so far) with HTTPD::Log::Filter. One-line parse-and-extract:

    perl -MHTTPD::Log::Filter -lne'BEGIN{$f = HTTPD::Log::Filter->new(form
+at=>"ELF",capture=>[qw(authexclude request referer)])} $f->filter($_)
+;printf "%s %s\n",$f->authexclude,$f->referer' /var/your_httpd_log_he
+re

    [download]

    Granted, I have a longer definition of "one-line" than some, but it still fits in my tcsh buffer...


    If God had meant us to fly, he would *never* have given us the railroads.
        --Michael Flanders

[reply]
[d/l]
      I also had good results with HTTPD::Log::Filter extracting the query strings from my access_log. 
      my $hlf = HTTPD::Log::Filter->new(capture => [ qw(request status) ]);
...
    next unless $hlf->filter($ac_line);
    next unless $hlf->status == 200;
    my ($method, $query, $http_version) = split(' ', $hlf->request);

      [download]
[reply]
[d/l]

Back to Seekers of Perl Wisdom