Hi mjacobson,

I hope you don't literally mean days ... for 21,000,000 lines it would take you more than 57,000 years (57494.8 years) to process the file.  Once.

Having said that, it's not clear that you would benefit from using threads.  You would still need to send the data in each line to the thread doing the processing, and then collect the result(s) from that thread afterwards, and since you've now got multiple threads each vying for the CPU, it's quite possibly going to be even slower than a single process working on it.  Depending on what exactly you're doing, of course.

Having said that, can you explain a little further exactly what processing you're trying to do on each line (and how long it actually takes)?

s''(q.S:$/9=(T1';s;(..)(..);$..=substr+crypt($1,$2),2,3;eg;print$..$/

In reply to Re: Processing large file using threads by liverpole
in thread Processing large file using threads by mjacobson

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.