Hi Parv,
Wish I could pay you! Sorry about your employment status! :-( Here is the Perl script that I created. This works for me and produces the output that I targetted for.

To start processing records by the quarter of the hour, for example, 14:00:00, 14:15,..., I checked for (($minute%15) !=0), right at the beginning of the subroutine count_recs.
#!/usr/bin/perl -w use strict; use Getopt::Long; use Time::Local; use Time::localtime; my $file="xaa"; my %increment = (); my $interval = 900; #An interval of 15 minutes my %MonthToNumber = (); @MonthToNumber{qw (Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec) +} = map { sprintf "%02d", $_; } (0..11); count_recs(); sub count_recs { my ($Day, $Month, $Year, $Hour, $Minute, $Second); my ($datetime, $get_post); my $dmytime; my $dateproc; my @dates = (); my @dates_mod = (); open (INFILE, "<$file") || die "Cannot read from $file"; while (<INFILE>) { ($datetime,$get_post) = (split / /) [3,6]; $datetime =~ s/\[//; ($Day,$Month,$Year,$Hour,$Minute,$Second)= $datetime =~m#^(\d\d) +/(\w\w\w)/(\d\d\d\d):(\d\d):(\d\d):(\d\d)#; next if ($get_post =~ /\.js$/ || $get_post =~ /\.gif$/ || $get_pos +t =~ /\.css$/ || ($Minute%15) != 00); push (@dates, $datetime); } #outer while # Calculate the time in epoch seconds in the subroutine convert_epoch. + foreach $dmytime (@dates){ my $calc_seconds= &convert_epoch($dmytime); push (@dates_mod, $calc_seconds); } foreach $dateproc (@dates_mod) { $increment{$dateproc}++; } # Print the headings in the output file open (OUTFILE, ">>access_log.out") || die "Cannot write to $file"; print OUTFILE " Date Range of Time Visual Pages per seco +nd(VPPS)\n"; print OUTFILE " Avg Max Min\n"; print OUTFILE "------------ ----------------- ------------\n"; &calculate_time(); print OUTFILE "\n\n\n"; $interval = 3600; &calculate_time(); close(OUTFILE); close(INFILE); } sub convert_epoch { my $localdate = shift; my ($epoch_Day,$epoch_Month,$epoch_Year,$epoch_Hour,$epoch_Minute,$ +epoch_Second)= $localdate =~m#^(\d\d)/(\w\w\w)/(\d\d\d\d):(\d\d):(\d\ +d):(\d\d)#; my $epoch_seconds = timelocal($epoch_Second, $epoch_Minute, $epoch_ +Hour, $epoch_Day, $MonthToNumber{$epoch_Month}, $epoch_Year-1900); return $epoch_seconds; } sub calculate_time { my $first = 0; my $start_time; my $end_time; my $time; my ($max, $min, $avg, $count); my $partial=0; my $col_ind; my $processed; my $tm; foreach $processed ( sort keys %increment ) { if ($first == 0 ) { $start_time = $processed; $end_time = $start_time + $interval; $time = $start_time; $min = $increment{$processed}; $max = 0; $avg = 0; $first++; } if ($processed < $end_time) { if ($increment{$processed} < $min) { $min = $increment{$processed}; } if ($increment{$processed} > $max) { $max = $increment{$processed}; } $avg+= $increment{$processed}; $count++; $partial = 1; }else { $avg=$avg/$count; my $tm = localtime($end_time); my $start = localtime($start_time); if ($interval == 900) { $col_ind = 'P'; }else { $col_ind = 'H'; } printf OUTFILE "%01s %02d/%02d/%04d %02d:%02d:00-%02d:%02d:0 +0 %03.2f %03d %03d\n", $col_ind, ($tm->mon)+1,$tm->mday, $tm->year+19 +00,$start->hour, $start->min, $tm->hour,$tm->min, $avg, $max, $min; $min = $increment{$processed}; $max = $increment{$processed}; $avg = $increment{$processed}; $count = 1; $start_time = $processed; $time = $start_time; $end_time = $start_time + $interval; $partial = 0; } } if ( $partial == 1 ) { $avg = $avg/$count; $tm = localtime($end_time); my $start = localtime($start_time); printf OUTFILE "%01s %02d/%02d/%04d %02d:%02d:00-%02d:%02d:00 %0 +3.2f %03d %03d\n", $col_ind, ($tm->mon)+1,$tm->mday, $tm->year+1900,$ +start->hour, $start->min, $tm->hour,$tm->min, $avg, $max, $min; } }

Pl. suggest any improvements you can think of. I know that I used the MonthToNumber (convert "Apr" to "04" and so on) as a Global. Thanks to you for that input.

Also, I used the hash %increment as a global. May be I could pass it as an argument to the subroutine calculate_time?

I needed to process two intervals of time, 15min and 1 hour. That's why I called the subroutine twice, with the interval used globally. Any thoughts?

Performance wise, I got a 100MB file processed in about 1min 5 secs.

My next step is to be able to process multiple input files. Currently, I hardcoded an input file. But, I would like to pass multiple log files as input and process them all.
-Andy

In reply to Re: Re: Re: Re: Re: Parsing of the web log file, access_log by Andy61
in thread Parsing of the web log file, access_log by Andy61

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.