in reply to Re: Re: Efficiently parsing a large file
in thread Efficiently parsing a large file

sort will create separate temp files anyway which it then merges together and pipes to you. You really aren't gaining much since you need almost as much disk space. Also, pipes can slow down IO for huge files. They are usually limited to a few KB of buffer so you get a lot of swapping (i.e. sort feeds some bytes to you then goes to sleep, you wake up and process them then go to sleep, then sort feeds some more, etc.).
  • Comment on Re: Re: Re: Efficiently parsing a large file