in reply to Re: How to improve speed of reading big files
in thread How to improve speed of reading big files

Small suggestion..I would get rid of both the map{} and the grep{}. Both of these things use anon lists and at the end of day, some @anon_list gets pushed onto @lines. A simple loop that doesn't use these constructs should be faster. 
while (<$fh>)
{
   if (/Running|Dump|FromCB|Update/o)
   {
      push ( @lines, filterLog($_) );
   }
}

[download]
Update: 
push @lines, grep { /Running|Dump|FromCB|Update/o && filterLog($_) } <
+$fh>;

[download]
Is a good idea, but don't think that it works. @lines needs to get the output of filterLog(). Grep is just a simple "filter", so this would need to be a map{} otherwise looks like you always get the value of <$fh> when the value of the grep statement is "true"?

Replies are listed 'Best First'.
Re^3: How to improve speed of reading big files
by BrowserUk (Patriarch) on Sep 18, 2009 at 08:11 UTC

    Agreed on both counts. The grep would have to become map not vice versa, and doing away with both is better yet.

    At that point, I'd probably unwrap the filter sub back into the while loop so the regexes can operate directly on $_ "legitimately".

    And at that point, you start to wonder how many times each set of files is reprocessed with different sets of user/date/start-time/stop-times?

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
      I think we are on the same page on the main idea. I was surprised about this sub() as I wouldn't have even thought of that idea! 
      sub filterLog() {
    s/ {2,}/ /g;
    s/^((?:\S+ ){3}).+?\[?I\]?:/$1/;
    s/ ?: / /g;
    s/ ACK (\w) / $1 ACK /;
....}

      [download]
      I think that this is the idiomatic Perl way to set the $_ variable to some value. It is possible to say: $_ =x;. It is possible to use $_ like in your sub above. But I think this is the idiomatic way to refer to a $variable as $_. 
      sub filterLog
{ 
    my $input = shift;
    foreach ($input)
    {
       s/ {2,}/ /g;
       s/^((?:\S+ ){3}).+?\[?I\]?:/$1/;
       s/ ?: / /g;
       s/ ACK (\w) / $1 ACK /;
    }
    # $input has been modified...
#.......
}

      [download]
[reply]
[d/l]
[select]

        The point was that he was already implicitly using $_ in a couple of places, so why not go the whole hog and avoid making copies of every line.

        The foreach scalar is a neat trick, but ultimately a do-once loop is more confusing to beginners than manipulating a global variable, which they tend to do naturally anyway before they learn better. There are many ways of improving the code in the CS sense, but the OP asked how to speed things up.

        Avoiding unnecessary copying is one way. Unwrapping the sub back into the main loop is another. Avoiding the allocations involved in generating multiple long lists, and lists of anonymous arrays, is probably the most effective first step.

        If you need optimise, then the only way is to profile properly and re-run the test after each change. Since we're only party to two subs, there's no way to know if this might be better done using a simple shell pipe chain. We also don't know whether each set of files is processed by once by a single set of query parameters; once each by several sets; frequently by many sets etc.

        Under some scenarios, loading the logs directly into a DB and querying that makes most sense. Under others, the overhead of loading a DB would outweight the gains. Without the full picture you can only attempt to answer the question as asked.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

In Section Seekers of Perl Wisdom