Re: Re: Simplifying my script...

Not only are you reading the file F by piping cat, but you're reading it once for every line of the input file.

Wouldn't this be the way to save on memory? If it's a large file then wouldn't you read it line by line? I've been told with smaller files it's ok to read the whole thing and process it but otherwise you want to read it line by line.

Am I thinking about this from the wrong angle?

Comment on Re: Re: Simplifying my script...

Replies are listed 'Best First'.
Re: Re: Re: Simplifying my script... by matija (Priest) on Apr 25, 2004 at 06:10 UTC
I think you are. If the file is small enough to keep in memory, keep it in memory. If it's too big to keep in memory, you should put the data into the database, and mark the appropriate fields as keys. Then the database will know how to skip to exactly the right record much faster than you could do it by reading every record. Remember, if the file is big, it takes a long time to read - and you're reading it each time you go through that loop.	[reply]

Replies are listed 'Best First'.

Re: Re: Re: Simplifying my script...
by matija (Priest) on Apr 25, 2004 at 06:10 UTC

If it's too big to keep in memory, you should put the data into the database, and mark the appropriate fields as keys. Then the database will know how to skip to exactly the right record much faster than you could do it by reading every record.

Remember, if the file is big, it takes a long time to read - and you're reading it each time you go through that loop.

[reply]