Typical language analysis goes with your Option #2. I think I'll submit a code snippet which I used on some texts to extend some Moby Project results.

I'm not sure what you'd accomplish with the Option #3. You say you want to find the most common substring, but then your #3 says you'd mask out all occurrences of the most common "word." Don't you need to solve the question before you can perform that work? And secondly, if you wanted the second-most-common substring, is masking out the topmost common substrings going to be what you want?

off the record, heretofore, the officer found that in the
theater of war, one hath need of a weatherproof theory of games
[download]

Now, remove substring 'he' and you won't find 'the'. Remove 'of' and you won't find 'off'. Is this effect intentional in your search? (Sorry, I couldn't contrive an example with overlapping 5-character snippets.)

--
[ e d @ h a l l e y . c c ]