They way I handled this for doing work on a 8 way beowolf cluster was to take the data set and split it into 8 chunks. This seemed easier than trying to deal 1 to each machine in order.

In your case either split your file into 2 files before forking then each process deals with a half or (if possible) have the second seek to halfway through the file and start there, and make sure the first ends just before the second started.

This should be much better than trying to synchronize each thread, if you do it that way then you don't gain nearly the performance of having 2 unsynchronized threads. (A lot of time would be spent waiting for the other thread to work).

As I'm thinking of it, In each thread have the parent do odd lines and then have the child do even lines. As long as you are not modifying the original file you should be fine. (same concept as above).

If you are going to modify the file then have each thread write it's own temproary file, then after the major processing is done (child exits), have the parent merge the output, based on how you divided the data set.

If you could describe the nature of your problem it would make most of my guessing, and ideas go away and the "right" way to do it would become clear.

Hope this helped!

In reply to Re: Process file simultaneously after forking by Helter
in thread Process file simultaneously after forking by bheckel

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.