in reply to Reduce RAM required

Maybe I'm not understanding your requirement, but it seems to me you could just read in one window's worth of sequences, shuffle them, and write them to the file. Then you could empty the window buffer and repeat until you reach EOF. This way, your memory will never grow larger than the window size.

The algorithm is a little trickier that way because you need to deal with sequences that overflow the window, but I don't think it should be too hard.

EDIT: After reading hdb's comment, I would also suggest you go that route. That seems even better than what I was describing.

πάντων χρημάτων μέτρον έστιν άνθρωπος.

Replies are listed 'Best First'.
Re^2: Reduce RAM required
by onlyIDleft (Scribe) on Jan 09, 2019 at 16:04 UTC

    I want to obliterate signal by introducing ample noise by random shuffling. But think about it - if I bring distant regions of the sequences together and then shuffle them (or shuffle them and then bring them together), then it is more of global rather than local shuffling - which is something I prefer. You agree with my approach?