From your minimal description, I am assuming this scenario: you have a 100+ MB file that you need to load into some other application. It is slow. You are trying to throw 'threading' at it to speed it up without knowing where the slowness is coming from.

Where is the bottleneck? You state that there are 10 million lines that you are reading. What is the total size of the file? At 100 chars per line, you would have a 1GB file. This can be slow to read, large to represent fully in memory, etc.

There are many things that can cause 'slowness'. Until you can characterize where the slowness comes from, hold off on picking a solution. Characterize the method used by your application to accomplish the task. Is it reading all of the data into memory and then doing something with it, or is it interleaving reads with processing? If interleaving, is it discarding information it no longer needs? Is it reading from and writing to the same physical disk, same I/O channel, saturating the network, etc?

If you are saturating a single resource, threading will probably not solve your problem, and may, in fact, add to it, since you now need to manage the threads as well. Disk, I/O, network, CPU, and Memory are some of the more common resources that can be saturated, but there are others as well. Figure out how to make those resources less saturated (or do other work instead of waiting to use the saturated resource - which may involve threading).

Update: Added last phrase indicating where threading may be useful

--MidLifeXis

In reply to Re: Multi threading by MidLifeXis
in thread Multi threading by devil_encoder

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.