Hi Monks!
I have a large text file, which is assembled of ~11000000+ records in the form of: 
>NAME1
SEQUENCE1
>NAME2
SEQUENCE2
>NAME3
SEQUENCE3

[download]

I then have my perl script which reads some results and then wants to extract only certain records from this huge text file. It could be 100 records, 10 records, 45 records etc, this is based on my results.
My question is, since I suppose it would be rather time consuming to read the file each time searching for the desired record:
(I) can Perl handle a hash of tha size? Like 11000000 keys and values? In that case, as I can understand, the script would have to create the hash each time, so it would be rather time-consuming, no?
(II) would it be more efficient to store these records in a Mysql database and then retrieve them based on the NAME for example? Can the database store that many items without trouble? I suppose yes?
Thank you!

In reply to Efficient way to handle huge number of records? by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.