Could you check/advise on my plan before I try to put the code together?

I want to write an efficient random line grabber. As efficient as possible assuming large files. Say 10,000,000 lines just for argument's sake. I don't believe there's any way to do this directly without reading the whole file (correct me, obviously, if that's wrong).

So an indirect route. Code outline:

Given: the delimiter is normal $/ and file is guaranteed to contain multiple records.

  1. -s the file.
  2. Open a filehandle to it.
  3. sysseek int( rand( -s _ ) ).
  4. sysread X bytes from that mark.
  5. If we ssyseeked 0 or buffer begins with $/, look forward, including new sysreads if necessary, to find another $/ or "eof" (0 bytes returned).
  6. Else we read backwards, building a string with the buffer, till we have ^(sysseek 0|$/). And if we didn't have an ending $/, we remember where we started off and we read forward till we get one.

Does this bias the results against the 0th or the -1st records? Good approach? Better one(s)? Already solved and my research skills need improvement?

Side question: reading a file that only contains ASCII as utf8 can't hurt the read, right? I want to open the file utf8 so it can grow to eventually contain special character data.

(footnote: checked How do I pick a random line from a file? and a few others and found nothing complete though an AM brought up the idea here).

As always, thanks!

In reply to Approach to efficiently choose a random line from a large file by Your Mother

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.