ferrency is probably right that preprocessing is the only practical way to achieve a flat distribution (i.e., for all n-letter words to pop up with the same frequency). I don't have a solution that you'd want to implement, but it's an interesting question: how would you do this without preprocessing?

Solution 1:
Seek to a random place and read forward until you find an n-letter word.
Result:
Each word appears with a frequency proportional to the distance in the file between the beginning of its line and the beginning of the previous n-letter word's line. (It's hard to imagine when this would give desirable results, but the results would be (technically) random.)

Solution 2:
Seek to a random place. If the next line contains an n-letter word, use it; otherwise, seek to another random place.
Result:
Each n-letter word appears with a frequency proportional to the length of its predecessor's line. (For /usr/dict/words, it's better than the first solution, but still not good.)

Solution 3:
1. Seek to a random place.
2. Eat a random number of lines (1-N) (flat dist.)
3. If the next word has n letters, use it; otherwise go back to step 2.
Result:
I think I'll implement this to see how well it works. After looking at /usr/dict/words, I doubt this will work much better than Solution 2 for N<100, but I'll have to see.

This seems like a very tough problem to me. I'm sure that preprocessing is the way to go here, but if there is an efficient way to do it with no preprocessing (either for /usr/dict/words or for an arbitrary file) it would be fascinating to see it.

P.S. I don't know if the original poster wants approximately equal probabilities or not; he/she will probably use preprocessing anyway. P.P.S. In the algorithms, I'm assuming you go to the beginning of the file when you hit EOF.

In reply to Re: RandomFile by grackle
in thread RandomFile by jettero

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.