I belive that the bottleneck in such a search is probably disk access. When I grep through between 10M and 100M of data, it seems that it's always much faster if I rerun the search, as the files are cached in memory at that time. Egrep is very fast, as it builds a dfa, so it's probably faster than perl; however, as you probably can't hold all 20G of data in the memory, the difference will be probably negligable.

Update: as a consequence, if you want to make grepping fast, you'll have to make disk access faster; make sure dma is set up correctly, put the 10 files on different disks if you can, copy the data to hard disk if they are on cd or dvd or nfs now. Also, do multiple searces at once if you can. (You're not zgrepping compressed files, are you? That would slow things down.)

Update 2: as for your update: to tell how much memory they are using, use top or ps if this is on a unix box.

In reply to Re: perl performance vs egrep by ambrus
in thread perl performance vs egrep by dba

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.