Having 4 processes (or as I would use: threads) reading from different parts of the same file is not a problem.

The problems come entirely from the hideous format design of FastQ files.

As Wikipedia puts it:

it can make parsing complicated due to the unfortunate choice of "@" and "+" as markers (these characters can also occur in the quality string).

Dividing the file size in to 4 and having each thread/process seek into the file to a different position is simple and fast.

The problem is how to then skip forward from the calculated start position to locate the start of the nearest (next) 4-line record.

The format specifies that the first character of the 1st line of each record is '@'; and the first character of the 3rd line is '+'; but dumbly, these marker characters can also appear as part of the quality information in the 2nd and 4th lines -- including as the first character of each of those lines.

That makes leaping into the file and find the start of a record surprisingly difficult, with the only simple alternative being to read forward in groups of 4 lines from the start of the file, which kinda defeats the purpose.

Theoretically, reading forward until you have 4 consecutive lines where the 1st & 3rd start with '@' & '+' respectively, and the other two do not, should (I think) establish a datum point from which an appropriate starting point can be calculated from which each thread/process can advance. I'll get back to you once I've convinced myself of that.

How about Tie::File? From what I understand, it does not store the file in memory, could be suitable?

We have a list of all possible regular expression matching a sequence header for the fastqs that we are working with. Say for a set of 4 lines, you would have:

@BLABLA:1:2:3:2
ACGTACGT...
+
DDDEEHFG...
[download]

So all headers are:

 ^@\S+:\d+:\d+:\d+:\d+\n

But headers are not of a fixed length, so how would you seek into a file if you need the byte start and end position?

How about Tie::File? From what I understand, it does not store the file in memory, could be suitable?

No, but it does have to read every line in the file in order to work out where every line starts, which makes startup (actually tieing the file) horribly slow with big files.

And you can do that yourself for 1/10 the time cost.

This is reading the 10 million lines in a 400MB FastQ file and storing their start positions in an array using standard Perl:

C:\test>perl -MTime::HiRes=time  
-E"my @lines; $t=time; $lines[ @lines ] = tell *ARGV while <>; print t
+ime - $t;" sample.FastQ
5.8082218170166
[download]

Takes 5 seconds

Doing the same thing with Tie::File takes 12 times longer:

C:\test>perl -MTime::HiRes=time -MTie::File -E"$t=time; tie my @lines,
+ 'Tie::File', 'sample.fastq'; say $#lines; say time-$t;"
9999999
60.3115701675415
[download]

And that was with the advantage of the file still being in the file cache. It takes much monger still from a cold cache.

The problem is that in order for that module to allow in-place editing of the tied file, the author had to jump through some pretty extraordinary hoops with internal caching and spilling algorithms.

Any how, I've tested my seek and start method and it works. Give me a few minutes to comment it a little and I post it.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

That last comment was by me but forgot to log in...