Re^4: multiple processes to access one file

How about Tie::File? From what I understand, it does not store the file in memory, could be suitable?

We have a list of all possible regular expression matching a sequence header for the fastqs that we are working with. Say for a set of 4 lines, you would have:

@BLABLA:1:2:3:2
ACGTACGT...
+
DDDEEHFG...
[download]

So all headers are:

^@\S+:\d+:\d+:\d+:\d+\n

But headers are not of a fixed length, so how would you seek into a file if you need the byte start and end position?

Comment on Re^4: multiple processes to access one file Select or Download Code

Replies are listed 'Best First'.
Re^5: multiple processes to access one file by BrowserUk (Patriarch) on Aug 21, 2012 at 07:29 UTC
How about Tie::File? From what I understand, it does not store the file in memory, could be suitable? No, but it does have to read every line in the file in order to work out where every line starts, which makes startup (actually tieing the file) horribly slow with big files. And you can do that yourself for 1/10 the time cost. This is reading the 10 million lines in a 400MB FastQ file and storing their start positions in an array using standard Perl: `C:\test>perl -MTime::HiRes=time -E"my @lines; $t=time; $lines[ @lines ] = tell ARGV while <>; print t +ime - $t;" sample.FastQ 5.8082218170166` [download] Takes 5 seconds* Doing the same thing with Tie::File takes 12 times longer: `C:\test>perl -MTime::HiRes=time -MTie::File -E"$t=time; tie my @lines, + 'Tie::File', 'sample.fastq'; say $#lines; say time-$t;" 9999999 60.3115701675415` [download] And that was with the advantage of the file still being in the file cache. It takes much monger still from a cold cache. The problem is that in order for that module to allow in-place editing of the tied file, the author had to jump through some pretty extraordinary hoops with internal caching and spilling algorithms. Any how, I've tested my seek and start method and it works. Give me a few minutes to comment it a little and I post it. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply] [d/l] [select]
Re^5: multiple processes to access one file by julio_514 (Acolyte) on Aug 21, 2012 at 07:04 UTC
That last comment was by me but forgot to log in...	[reply]

And you can do that yourself for 1/10 the time cost.