Lets say he has split the huge file into 10 parts and he runs 10 threads. Using your schema, he would require one pipe for each of the 600 output files in each of the 10 threads; and another 600 threads running select loops to coalesce the records and write them to the 600 output files. So 610 threads and 6000 pipes!

That is not how I see it being done, and you clearly misunderstand what the pipes are doing, and as usual you use incorrect math to make the problem seem worse, somehow hoping it will prove your point. Not today, my man.

First there would only be 10 pipes, if there were 10 threads. The main thread can open as many disk files as it wants, and as the data comes into the select loop, some logic can be applied to determine which of his 600 output files it gets written to. The select loop would act as a central collection point/filter, allowing one piece of logic to decide where any thread output should be ultimately be written.

Its simple: the threads write to the select loop, the select loop decides where the particular data should be written. Otherwise he would need to incorporate the logic for which file to output to, into each thread, AND THEN have to worry about flocking those files. My method avoids that flocking problem by making the select loop the master disk writer and filter.

I'm not really a human, but I play one on earth.
Old Perl Programmer Haiku ................... flash japh

In reply to Re^11: how to split huge file reading into multiple threads by zentara
in thread how to split huge file reading into multiple threads by sagarika

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.