in reply to Simplifying my script...

Not only are you reading the file F by piping cat (what on earth posessed you to do that?), but you're reading it once for every line of the input file.

You should just read it once, do the split, and store each line as a row in an array of arrays. That should speed your script up considerably.

Replies are listed 'Best First'.
Re: Re: Simplifying my script...
by u235sentinel (Hermit) on Apr 25, 2004 at 02:14 UTC
    Not only are you reading the file F by piping cat, but you're reading it once for every line of the input file.

    Wouldn't this be the way to save on memory? If it's a large file then wouldn't you read it line by line? I've been told with smaller files it's ok to read the whole thing and process it but otherwise you want to read it line by line.

    Am I thinking about this from the wrong angle?

[reply]
      I think you are. If the file is small enough to keep in memory, keep it in memory.

      If it's too big to keep in memory, you should put the data into the database, and mark the appropriate fields as keys. Then the database will know how to skip to exactly the right record much faster than you could do it by reading every record.

      Remember, if the file is big, it takes a long time to read - and you're reading it each time you go through that loop.

[reply]

In Section Seekers of Perl Wisdom