Your skill will accomplish what the force of many cannot |
|
PerlMonks |
Re: Re: Need to process a tab delimted file *FAST*by clintp (Curate) |
on Mar 03, 2004 at 13:56 UTC ( [id://333540]=note: print w/replies, xml ) | Need Help?? |
If the I/O isn't your problem then consider the following:
* Don't use Perl. Re-write this in C, keep your keys in a tree of some kind (determined by the distribution of the keys) and the values (tot, max, last) in the tree. It's always faster to use a specifically designed tool (a single-purpose, well written C program) than a general purpose multitool (like Perl). * If you have any control over the input stream: instead of tab-delimited data, use space/column-delimited data. It's (probably) faster to extract two substrings of known width than to split. Leave the whitespace on the keys when storing, remove it later when redisplaying (if necessary).
In Section
Seekers of Perl Wisdom
|
|