Hi Fellow monks, i have a huge txt file in the 1XXMB size, and i need to go loop through the whole file extracting line by line data. I've tried a few times and currently it takes about one minutes plus or so to extract the whole thing.

my algorithm is a simple one as i just open up the file, loop through it, regex match it and extract it out by directly print OUT to the output file as the memory cannnot hold the whole file at all once.

i have a limit file which tells my script what to extract so that i don't have to extract everything.the limit file contains of test names which i can use to regex match in the txt file.

what i want to do here is to find a way to speed up the extraction time, i'm thinking of using multiple threads so that each thread will only extract a few test names specified in the limit file into a few output files. Is this a good idea? is there any better way to do this? thanks in advance~

In reply to How to speed up/multi thread extract from txt files? by MelaOS

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.