in reply to unique sequences

Here is a page from BioPerl showing how to get kmers from a large data set http://bioperl.org/howtos/Scrapbook_HOWTO.html#item24

Not a trivial problem and involves threads.