how do. This is part Perl-specific, part "general program theory" I am not by any stretch of the imagination a gene-e-yuss hacker, and I'm a little at sea as to where to start. Basically, I'm inheriting a system of scripts that parses an Apache logfile (rotated monthly, on a site that gets 6000+ hits / day; near the end of a month, we're looking at ~ 200MB) to generate various stats. Currently, the logfile gets split, analyzed, and various fragments of the data are stored in tab-delimited flat files. The aim is to have a representation of the server log data that will allow cross-referencing on various things (e.g. what was the most common request from UNIX clients in .de ?) The usual means of access is through a CGI script. Webalizer produces reports of the right sort, but the boss-man wants to keep this a custom job so we can maintain it in-house.

I take it that putting things in an SQL database is probably the sanest way to go here (faster at any rate, and it will certainly simplify things), but assuming that's not a goer (I'll make it one if I can; my supervisor seems mildly enthusiastic), does anybody have any advice concerning program design -- I think I like going OO for some parts (since the script generates various reports, I could have a master Report class and have various report types inherit from that one) but the reports are usually generated on the fly for easy web access. Would there be a significant performance hit going OO? (Disk space is not super-tight, but it's far from unlimited)
I'm having trouble looking up resources for this project; I guess in the main that's what I'm fishing for, since I don't have a CS degree and I'm not really up on various means of data representation =)

Assuming I can't get SQL, would I see a performance increase if (e.g.) I moved to DBM from the tab-delimited flat files? And if so, which DBM would you Monks recommend (from my glance at the capabilities, I suppose Berkeley DBM, but I have no experience with this? thanks for any help you can give!-

In reply to Dealing with large logfiles by arturo

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.