Think about Loose Coupling | |
PerlMonks |
Re: performance of File Parsingby roboticus (Chancellor) |
on Jul 07, 2011 at 10:55 UTC ( [id://913159]=note: print w/replies, xml ) | Need Help?? |
Based on your question, I'd propose a different method: Method 3: Parse the data as it's available: You can use File::Tail to open the file and read the data (even while another program is generating the file). This allows you to continuously read / parse / write. Thus, you can begin processing your data before you have the full terabyte. For example, suppose we use the following to generate a stream of data:
Then we can use something like this to read and parse the data while the original is running:
Then, when I ran them, the output of stream_read.pl was:
...roboticus When your only tool is a hammer, all problems look like your thumb.
In Section
Seekers of Perl Wisdom
|
|