Re^5: storing hash in temporary files to save memory usage

The indicated lines below are your keys for matching:

data1.txt
@NS500278
AGATCNGAA                           ********
+
=CCGGGCGG   1

@NS500278
TACAGNGAG                           ********
+
CCCGGGGGG   2
[download]

but they are 150 chars long? (Even though you've shown them as 9!?)

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re^5: storing hash in temporary files to save memory usage Download Code

Replies are listed 'Best First'.
Re^4: storing hash in temporary files to save memory usage by Anonymous Monk on Sep 21, 2016 at 06:51 UTC
yes they are 150 characters long but for simplicity of the example and for others to understand I have shown them as 9 but the length should not be fixed because if I try my code with some other data it should also work then.	[reply]
Re^5: storing hash in temporary files to save memory usage by AnomalousMonk (Archbishop) on Sep 21, 2016 at 13:49 UTC
Please take a look at some of the (many) posts by BrowserUk around this place that deal with Big Data processing, then carefully consider if you may not want to trim your requirements a bit. Give a man a fish: `<%-{-{-{-<`	[reply] [d/l]
Re^5: storing hash in temporary files to save memory usage by BrowserUk (Patriarch) on Sep 21, 2016 at 13:01 UTC
but the length should not be fixed because if I try my code with some other data it should also work then. Oh. Okay. Good luck with finding your fully generic solution. (Why not see if it can cure world hunger at the same time.) When you find the other solutions taking days or weeks and you are prepared to lower your expectation, come on back. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]