This appears to be more a miss-understanding the problem than a very bad script.

As far as I can see you are not rotating your logs therefor your script is scanning EVERY HIT your website has EVER had before it can analyse the last ten minutes.

Rotating logs is vital if you wish to do any kind of serious log analysis. My current record is having to rotate logs every 6 hours on a single website as files grow at over 1Mb/min. Multiple vhosts should always log to different files which you appear to be doing otherwise analysis takes a lot longer.

I would personally ask why you feel the need to write your own web based analysis system as Analog would probably do the job. If you do feel the need rotating logs daily can be a boon as you then know exactly which logs are needed for a particular set of days from the file name.

One method I have used to generate such realtime reporting is to use a daemon to run short periodic reports and store the stats in a database. Your script then processes the summary stats and not the raw log files.

Done correctly you can get very impressive results with these methods. I have had a system happily doing analysis on over 100M hits in sub 5 seconds but you need to really understand every possible shortcut.

However on the script side.. when counter=31 you have no counter increments happening. so you have the file be opened and reopened.

Hope it help
UnderMine

In reply to Re: Speed by UnderMine
in thread Log munging optimization tips? by mkent

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.