Hi all, I have a question to ask about rapid text searches in perl. I have a text format file as follow with 3 columns. The text file can be very huge, exceeding 1.5GB of size. Currently I'm using grep to look for the data I need. Let say I need to know the pair for 1101077781160, I would use grep to look for it. However, when the input is getting bigger, let say 1000 of entry to be grep, the process become very slow!

The question is if I'm going to use perl to store the text into hash, can I rapidly increase the search time?

And if I only looking for a single entry, perl script will still load everything into memory before it could look for the string I want?
Thank you for any input, would really like to know how can I speed up the text searches... thank you! 
1101077781160   1101077783656   bothChaff
1101077781161   1101077783657   bothChaff
1101077781162   1101077783658   bothChaff
1101077781163   1101077783659   bothChaff
1101077781164   1101077783660   oneChaff
1101077781165   1101077783661   oneChaff
1101077781166   1101077783662   bothChaff
1101077781167   1101077783663   bothChaff
1101077781168   1101077783664   bothChaff
1101077781171   1101077783667   bothChaff
1101077781172   1101077783668   bothChaff
1101077781173   1101077783669   bothChaff
1101077781175   1101077783671   bothChaff
1101077781176   1101077783672   bothChaff
1101077781177   1101077783673   bothChaff
1101077781179   1101077783675   bothChaff
1101077781180   1101077783676   oneDegen
1101077781181   1101077783677   bothChaff
1101077781182   1101077783678   oneChaff
1101077781184   1101077783680   bothChaff

[download]

In reply to Rapid text searches by joomanji

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.