From: Uri Guttman <uri@stemsystems.com>
Message-ID: <x77it0drya.fsf@mail.sysarch.com>
Date: Thu, 29 Mar 2007 09:50:37 -0500

>>>>> "cc" == cadetg@googlemail com <cadetg@googlemail.com> writes:

  cc> Dear Perl Monks, I am developing at the moment a script which has to
  cc> parse 20GB files. The files I have to parse are some logfiles. My
  cc> problem is that it takes ages to parse the files. I am doing something
  cc> like this:

this isn't perlmonks. that is a web site community. this is usenet.

  cc> my %lookingFor;
  cc> # keys => different name of one subset
  cc> # values => array of one subset

  cc> my $fh = new FileHandle "< largeLogFile.log";

FileHandle is an old and deprecated module. where did you learn to use
it? you can just do basic open as you don't need any OO here. also
ALWAYS check for errors on opens.

  cc> while (<$fh>) {

don't use plain while to read lines. the proper idiom is:

	while( my $line = <$fh> ) {



  cc>   foreach my $subset (keys %lookingFor) {

that makes no sense. keys are strings. they are not array refs. yet the
line below dereferences the key into an array. which means you are not
using strict and you are testing for wrong stuff.

  cc>     foreach my $item (@{$subset}) {

why the two loops? since you always look at all the items for each line,
factor out the items list to before the loops:

	my @items = map @$_, values %lookingFor ;

  cc>       if (<$fh> =~ m/$item/) {

that is very wrong. it reads a new line in for each match in the
subset. the line from the while is in $_ at the moment. 

  cc>         my $writeFh = new FileHandle ">> myout.log"; print $writeFh <
  cc> $fh>;

why are you opening the output file EACH time you want to print? just
open it before the loops and print to it. my previous advice on opens
applies here too.

  cc> I've already tried to speed it up by using the regExp flag=>o by doing
  cc> something like this:

forget speed for now. your code is broken. are getting any useful output
at all? i can't see how as you are looking for stuff with broken code.

uri

-- 
Uri Guttman  ------  uri@stemsystems.com  -------- http://www.stemsystems.com
--Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
Search or Offer Perl Jobs  ----------------------------  http://jobs.perl.org