Hi all,
I am new to this group. Found very good information!

Here is what I am looking at.

I am creating a perl script to parse the web log file, access_log.

The format of the access_log is:

127.0.0.1 - - 15/Jun/2003:13:54:02 -0100 "GET /xxxx HTTP/1.1" 200 34906

The goal is to

1. Perfom a count of the pages for the given timestamp. It is possible that multiple pages exist with the same timestamp (As the timestamp I mentioned above).

2. Within a range of time interval, say, 15 minutes starting with the timestamp of the first line in the log file, I would like to compute the average of the number of pages, minimum and maximum number of pages in that interval.

3. I would like the output as below. Following is just an example.

Time Average Pages Min Pages Max Pages

--------------------------- ----------------- -----------------

15/Jun/2003:14:09:02 6.5 3 10

15/Jun/2003:14:24:02 5.5 4 7

----------

Here is the perl script I created.

----------------
#!/usr/bin/perl 

###use strict;

use Getopt::Long;
use Time::Local;

my $file="access_log_modified";
my $begin_time = "";
my $end_time;
my @visual_pages = ();
my @final_visual_pages = ();
my %increment = ();
my ($datetime, $get_post, $Day, $Month, $Year, $Hour, $Minute, $Second
+);
my $interval = 60;  #An interval of 1 minute

count_recs();

sub count_recs {

   open (INFILE, "<$file") || die "Cannot read from $file";
   WHILELOOP: while (<INFILE>) {
                 ($datetime,$get_post) = (split / /) [3,6]; 
                 $datetime =~ s/\[//;
                 ($Day,$Month,$Year,$Hour,$Minute,$Second)= $datetime 
+=~m#^(\d\d)/(\w\w\w)/(\d\d\d\d):(\d\d):(\d\d):(\d\d)#;

                next WHILELOOP if ($get_post =~ /\.js$/ || $get_post =
+~ /\.gif$/ || $get_post =~ /\.css$/);
                    
                   unless ($begin_time) {
                      $begin_time = $datetime;
                   }                    
                   push (@dates, $datetime);
                         
   } #outer while

   foreach $dateproc (@dates) {
      $increment{$dateproc}++;
   }
   foreach $dateproc (sort keys %increment) {
      push (@{$processed_visual_pages{$dateproc}}, $increment{$datepro
+c});
      print "$dateproc @{$processed_visual_pages{$dateproc}}\n";
   } 

   close(INFILE);
}

[download]

----------

Here is the output I get:

---------------

25/Apr/2003:13:54:02 3

25/Apr/2003:13:54:19 2

25/Apr/2003:13:54:22 4

25/Apr/2003:13:54:34 3

25/Apr/2003:13:54:38 5

-----------------

I am able to get the count for each of the timestamps. However, I am having trouble getting the records formatted in the interval range.

I shall appreciate an early help in solving this problem.

Thanks in Advance

Andy

In reply to Parsing of the web log file, access_log by Andy61

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.