Well Browser, it seems that you were right and my line tracking idea didn't work out so well. Even with your more efficient code, it just took forever. Grep was very fast, but the line tracking combined with outputting certain lines of the file was extremely slow. What I ended up doing was a combination of grep (using the 50 line context option), and a foreach loop on the resulting array data. I parsed the data that way and it's very fast. On that slow P4/512MB machine I can parse a 1GB file for a fixed string in about 15 seconds. The same parsing done completely with a Perl file loop takes takes 15x as long.

I can only speculate but Perl needs to read the entire file line by line and search for the string, while grep obviously does something far more efficient to find the string. In the end, I haven't yet found anything close to the speed of grep to parse large 1GB files.

If anyone out there can find a faster way, I would be impressed. Perhaps there is a Perl module that can do very fast searches on files? AWK? I think that's comparable in terms of speed to grep, and I could also parse the data with it... hmmm..

In reply to Re^5: search array for closest lower and higher number from another array by bigbot
in thread search array for closest lower and higher number from another array by bigbot

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.