Re^2: Deconvolutinng FastQ files

Just to make the point about the inappropraitness of Tie::File for this.

Using normal IO on a 400MB fastQ file takes 20 seconds:
And use 1.3 MB of ram; performs 96e3 reads 97e3 writes; and consumes 42 billion clock cycles.
Giving a projected runtime for the OPs 34GB file of 30.8 minutes.
Using Tie::File on that same 400MB file takes 1340 seconds:
And uses 601MB of ram; performs 10000e3 reads; 95e3 writes; and consumes 3,182 billion clock cycles.
Giving a projected runtime for the OPs 34GB file, of 34 hours!

Tie::File has its uses. This is not one of them.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re^2: Deconvolutinng FastQ files

Replies are listed 'Best First'.
Re^3: Deconvolutinng FastQ files by Kenosis (Priest) on Aug 06, 2012 at 21:34 UTC
I'll remember: `my %mindHash = ( 'Tie:File + Large File' => '"Run Away!" (King Arthur, Monty Py +thon)' );` [download]	[reply] [d/l]
Re^4: Deconvolutinng FastQ files by afoken (Chancellor) on Aug 07, 2012 at 05:23 UTC
`my %mindHash = ( 'Tie:File + Large File' => '"Run Away!" (King Arthur, Monty Py +thon)' );` [download] Oh noes! Kenosis has deleted all of his memory except for the fact that Tie::File and large files are to be avoided. `$mindHash{'assign to entire hash'}='only if you want to delete all exi +sting data';` [download] Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply] [d/l] [select]
Re^5: Deconvolutinng FastQ files by ww (Archbishop) on Aug 07, 2012 at 13:56 UTC
But he'll never feel the loss ... unless he reads your node.	[reply]
Re^5: Deconvolutinng FastQ files by Kenosis (Priest) on Aug 07, 2012 at 15:18 UTC
This is good! (However, I may have just increased my memory, since the word Kenosis means emptiness... :)	[reply]