Hi all,

I recently came across a speed problem while reading big files (around 1 million lines). The whole process is taking about 100sec i'd like to know if i can speed it up.

I'm using the code below where :
- mergeLogs is used to retrieve only the relevant lines (76 sec)
- filterLog is used to filter lines (23 sec)
- openLogFile returns a handle on the file (0.2 sec)
- The map/sort/map combo is used to sort the lines by time

Is there any way to speedup the process ?

sub mergeLogs()
{
    my $day = shift;
    my @files = @_;
    my @lines;

    foreach my $file (@files)
    {
        my $fh = &openLogFile($file);

        if (! defined $fh)
        {
            warn "$0: ignoring file $file\n";
            next;
        }

        warn "-> processing $file\n" if $opts{'verbose'} > 0;

        push @lines, map { &filterLog($_) } grep { /Running|Dump|FromC
+B|Update/o } <$fh>;
        close $fh;
    }

    @lines =    map { $_->[0] }
                sort { $a->[1] cmp $b->[1] }
                map { [ $_, /(\d{2}:\d{2}:\d{2}\.\d{3})/o ] } @lines;

    return \@lines;
}

sub filterLog()
{
    my $line = shift;

    $line =~ s/ {2,}/ /g;
    $line =~ s/^((?:\S+ ){3}).+?\[?I\]?:/$1/;
    $line =~ s/ ?: / /g;
    $line =~ s/ ACK (\w) / $1 ACK /;

    return unless ! exists $opts{'day'} || /^$opts{'day'}/o;
    return unless ! exists $opts{'user'} || /[\(\[]\s*(?:$opts{'user'}
+)/o;

    if (exists $opts{'start-time'} || exists $opts{'stop-time'})
    {
        if ($line =~ /(\d{2}:\d{2}:\d{2}\.\d{3})/o)
        {
            return unless ! exists $opts{'start-time'} || $opts{'start
+-time'} lt $1;
            return unless ! exists $opts{'stop-time'} || $opts{'stop-t
+ime'} gt $1;
        }
    }

    warn $line if $opts{'verbose'} > 3;

    return $line;
}

[download]

In reply to How to improve speed of reading big files by korlaz

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.