Re: Re: Efficiently parsing a large file

don't forget you could just read the output of sort directly.. that way you don't actually need to create another file to read in..

Jason L. Froebe

No one has seen what you have seen, and until that happens, we're all going to think that you're nuts. - Jack O'Neil, Stargate SG-1

Comment on Re: Re: Efficiently parsing a large file

Replies are listed 'Best First'.
Re: Re: Re: Efficiently parsing a large file by bluto (Curate) on Apr 08, 2004 at 21:33 UTC
sort will create separate temp files anyway which it then merges together and pipes to you. You really aren't gaining much since you need almost as much disk space. Also, pipes can slow down IO for huge files. They are usually limited to a few KB of buffer so you get a lot of swapping (i.e. sort feeds some bytes to you then goes to sleep, you wake up and process them then go to sleep, then sort feeds some more, etc.).	[reply]

Replies are listed 'Best First'.

Re: Re: Re: Efficiently parsing a large file
by bluto (Curate) on Apr 08, 2004 at 21:33 UTC

sort will create separate temp files anyway which it then merges together and pipes to you. You really aren't gaining much since you need almost as much disk space. Also, pipes can slow down IO for huge files. They are usually limited to a few KB of buffer so you get a lot of swapping (i.e. sort feeds some bytes to you then goes to sleep, you wake up and process them then go to sleep, then sort feeds some more, etc.).

[reply]