Good Morning all you Enlightened Monks,

I'm trying to read a pattern from a quite fast logfile (writes approximately 4Megs of text in one minute and rotates every 10 minutes). I am using the File::Tail Module to keep the file in "constant tail". Everytime the code finds a set of patterns, they should be stored within a table on an Oracle db. Here is my code snippet and below some of the questions i'd like to be enlightened on: 
use File::Tail;

my $file_name = qw( logfile.log ) ;
my $tail_file = File::Tail->new( name => $file_name, resetafter => 3, 
+adjustafter => 5 ) ;
my $tail_line ;

my %pattern = (
    "Pattern1" => "Short Pattern to put on DB 1",
    "Pattern2" => "Short Pattern to put on DB 2",
    "Pattern3" => "Short Pattern to put on DB 3",
) ;

while ( defined($tail_line = $tail_file->read) ) {
    foreach my $key ( keys %pattern ) {
        if (  $tail_line =~ /$key/ ) {
            --->> insert "Short Pattern" into the DB 
        }
    }
}

[download]
My questions are the following:

1. Is this the best way to grep the pattern on such a fast file? Can there be possible data loss using this module?
2. What could be the best way to put the data on the database? I was thinking of a single connection that stays open all time or eventually put the data on a textfile and handle with a sqlldr crontabbed every minute.
3. This executable should be lightweight and robust for the system it runs on. Could there be better approaches to solve this issue?

Thanks,

Francesco

In reply to Reading from a fast logfile and storing on Oracle by longjohnsilver

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.