Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

What I would like is a script that will tail the access log of my web server for a given number seconds. Then I can run this script once an hour or at a given time to see how many pages are being loaded per second.

I tried using alarm to do something like this but it doesn't seem to work as I had hoped. Here's what I've tried so far. 

#!/usr/bin/perl
open ACCESS, "|tail -f /var/log/apache/access.log";
eval{
   alarm(10);
   while(<ACCESS>){
      print;
   }
};
close ACCESS;

[download]
Thanks, Aaron

Replies are listed 'Best First'.
(ar0n: our hero CPAN) Re: Tailing the access log
by ar0n (Priest) on Mar 30, 2001 at 01:02 UTC
    Person 1: Look, up in the sky!
    Person 2: Is it a bird?
    Person 1: Is it a plane?
    Person 3: No, it's CPAN and he's coming our way!

    CPAN descends and lands on the ground, just a few meters in front of the crowd, and walks into an office building. He walks up to a Perl programmer, and standing behind his back he says:

    CPAN: I suggest you use File::Tail.

    Programmer: But of course! I totally forgot to check! Thanks CPA...

    But before the programmer could turn around, our hero flew off through an open window to assist another Perl programmer in distress...

    [ar0n]

[reply]
Re: Tailing the access log
by Masem (Monsignor) on Mar 30, 2001 at 00:43 UTC
    I dunno on that solution, but if all you are trying to do is get a rough estimation of the number of served docs/sec, let me suggest an alternative method, easily doable in perl:

    Grab the last n lines via tail from the access log and pipe it into a filehandle as you have it now (no -f, of course). 'n' would be based on how active you believe your server is; if you get 100,000 hits a day, on average, n should be around 100 (a hits per minute estimate would be N / 1440, N being the hits per day).

    Grab the first time string on the date, and store it.

    Subtract that date from localtime (using something like Date::Manip), to get a number of minutes, t_min.

    n / t_min should be an accurate measure of the # of requests served per minute.

    If you want to strictly limit it to docs, run through the n lines that you get back from tail, and increment a counter for each type of doc that you really want (eg look for /\.html/), call that counter x. x / t_min would be the number of html files served per minute then.

    This should be sufficiently fast that you can run it every minute or so without any major slowdowns and get a good running stat on your averages.

    Dr. Michael K. Neylon - mneylon-pm@masemware.com || "You've left the lens cap of your mind on again, Pinky" - The Brain
[reply]
[d/l]
Re: Tailing the access log
by runrig (Abbot) on Mar 30, 2001 at 00:49 UTC
[reply]
Re: Tailing the access log
by tinman (Curate) on Mar 30, 2001 at 01:01 UTC

    As a follow up to what Masem proposed, this is a code snippet that I used to monitor my Squid cache performance: 

    #!/usr/bin/perl 

package DateParser;

use Date::Manip;
sub date_mangler {
        $arg = $_[0];
        $times = $_[1];
        $seconds_gm = time();
        $some_day = localtime($seconds_gm);
        $parsed=&ParseDate($some_day);
        $epoch = &UnixDate($parsed,"%s");
        if(not defined $times) { $times = 1; }
        if(not defined $arg) { $arg = 'default'; }

        if(($arg eq '-min') or ($arg eq '-MIN')){
                $epoch -= ($times *60);
        }
        elsif(($arg eq '-hour') or ($arg eq '-HOUR')){
                $epoch -= ($times * 3600);
        }
        elsif(($arg eq '-day') or ($arg eq '-DAY')){
                $epoch -= ($times * 86400);
        }
        else {
                $epoch -= 86400;
        }
        return $epoch;
}
1;

    [download]

    A little background: Squid uses epoch time (seconds since epoch) as the date format. Calling DateParser::date_mangler (I tried hard to think of a better name, I promise ;) gives you the minumum number of seconds for the date metric (minute, hour) passed in as a parameter.

    Thereafter, simply pick all the lines that have a date stamp larger than the one returned from this function..
    HTH, and as always, comments on code much appreciated..

    Update:File::Tail is exactly what you want for your job, as noted by others. My procedure is used to do analysis on static files that have already been rotated.

[reply]
[d/l]
Re: Tailing the access log
by Dominus (Parson) on Mar 30, 2001 at 08:28 UTC
    There's a discussion of tailing in perlfaq5. You might want to take a look.
[reply]
Re: Tailing the access log
by physi (Friar) on Mar 30, 2001 at 12:58 UTC
    Well we are doing a similar job in this way:
    fork && exit;           ### making a daemon
open FILE, "ACCESS";
for (;;) {
 while (<FILE>){
   $lines++;
 }
print "$lines to somewhere, maybe to another file";
$lines=0;
sleep 3600;
}

    [download]
    This script works as a deamon.
    You can write the numbers to another file,
    or maybe you can blow this script off, to generate a new webpage with all the statstics you need :)

    -----------------------------------
--the good, the bad and the physi--
-----------------------------------

    [download]
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom