Hi there, I've written a fairly simple script which reads through one master list line by line and compares entries to other lists in other files, returning essentially a count of matches. The problem is, my comparison lists are rather large (100's of millions total) and each entry needs to run through each one. I have access to a large multi-core server, but I don't know how to modify my code to utilize more processors. I want to do a "work crew" thread model with each thread running a portion of the master list simultaneously and then concatenating the results at the end.

My elementary understanding of this is that I would turn my process into a subroutine with parameters dictating which list entries it will work on, then invoking new threads with the threads module to handle all sections of the master list. Does this sound like the right approach? I just want to know if there are any obvious problems with my plan of attack, or any major things I'm missing since I'm a complete multi-threading newbie.

Also, will perl automatically run these different threads on different cores, or is there something else I need to do to make that happen? I'm running perl on RHEL6 by the way.

In reply to multi threading by egunth

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.