in reply to Storing unordered data from file in memory

Now my question is how to do it. Is the best way to go once through the input file, sort it by address and then do it as before? But I think that this will have a slow performance. Here you see only a small example. The file can be really huge (e.g. 200 MB)
If the file is huge, I'd use sort(1) (that is, the external command) to sort the file. One may want to use grep to filter out unwanted lines (those not starting with S3). You may want to use a filter to put the address part into a column for sorting.

One of the advantages is that sort(1) doesn't easily run out of memory (although, with modern computers, 200MB should be ok) - it will resort to using temp files if it doesn't have enough memory to do it all in-memory.

  • Comment on Re: Storing unordered data from file in memory

Replies are listed 'Best First'.
Re^2: Storing unordered data from file in memory
by Dirk80 (Pilgrim) on Jul 21, 2010 at 13:05 UTC

    Is there also a perl sort command for files? I've seen that there exists File::Sort. What do you think of this? I want that the script is independent of OS and independent of the availability of an external command.

    A reply falls below the community's threshold of quality. You may see it by logging in.