My initial though as to use different processes, and have each process read in a different part of the file.

But it seems you're only interested in every 4th line, and while it's fairly easy to find lines when starting to read from the middle, you cannot know which line will be a 4th one without having counted from the beginning.

So, perhaps you should first look where the bottleneck is. CPU? IO? If it's CPU bound, using threads or forks may improve things. If it's IO bound, then using multiple threads/processes using different controllers may help. But only if they read different parts of the file, and then you're back to the sync problem. And then there are still many factors that play a role (one disk, multiple disks, mirrors, striping, disk/controller caches, other IO, etc) determining how much there's to gain.

In reply to Re: buffering from a large file by JavaFan
in thread buffering from a large file by cedance

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.