Has anyone here who is claiming that perl can't outrun grep actual run the script that I posted here that dws wrote? This dws guy is on to something. I was finally able to modify it to work like grep and it's 14 time faster than grep. I am working with a 484 MB mailog.

This could be more elegant but this my rookie solution.. 

while ( $window =~ m/([a-zA-Z]{3}\s{1,2}\d{1,2}.*\n)/oigc  ) {
            $line = $1;
            if ( $1 =~ /$re/ ) {
                &$callback($line);
            }
        }

[download]
ls -ltrh /var/log/syslog-ng/server2/ | grep maillog.2
-rw-r----- 1 root logs 484M Mar 11 11:13 maillog.2
-rw-r----- 1 root logs 230M Apr  1 04:10 maillog.2.gz

[dmathis@aus02syslog ~]$ date; ./jujuspeed; date
Thu May  1 19:27:57 CDT 2008

Feb 28 09:53:49 exmx2 sendmail[XXXXX]: 8791: to=<hidden@hotmail.com>, 
+delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=X3604, relay=mx1.h
+otmail.com. [X5.5X.2X5.X], dsn=2.0.0, stat=Sent ( <X4X0399.120421402X
+XXX.JavaMail.root@hidden.com> Queued mail for delivery)

Thu May  1 19:28:10 CDT 2008

Time taken: 13  Seconds


[dmathis@aus02syslog ~]$ date; egrep -i 'hidden@hotmail.com' /var/log/
+syslog-ng/server2/maillog.2; date

Thu May  1 19:28:48 CDT 2008

Feb 28 09:53:49 exmx2 sendmail[XXXXX]: 8791: to=<hidden@hotmail.com>, 
+delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=X3604, relay=mx1.h
+otmail.com. [X5.5X.2X5.X], dsn=2.0.0, stat=Sent ( <X4X0399.120421402X
+XXX.JavaMail.root@hidden.com> Queued mail for delivery)

Thu May  1 19:31:57 CDT 2008

Time Taken: 189 Seconds

[download]

Thanks for all of the help on here. I have learned alot :)

In reply to Re: Matching lines in 2+ GB logfiles. by Anonymous Monk
in thread Matching lines in 2+ GB logfiles. by dbmathis

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.