MajinMalak:

There's nothing magical about 50. Choose a larger number so you can zip through the file faster. Say 100,000 for instance.

Note: One difficulty that you'll want to be aware of is that you may break a line right in the middle of where your regexp would match, causing you to miss the data. One fix for that is that when you read the next buffer, keep the last X characters of the previous buffer and append it to the beginning of the string before trying the match. Something like:

$/ = \100000;
my $prev = '';
while (<$FH>) {
    my $line = $prev . $_;
    if ($line =~ /a funky regex/) {
        do some work here;
    }
    # Adjust 100 upwards if regex can match something longer than 100 
+chars
    $prev = substr($line,-100);
}

[download]

...roboticus

When your only tool is a hammer, all problems look like your thumb.

In reply to Re^5: Out of Memory - Line of TXT too large by roboticus
in thread Out of Memory - Line of TXT too large by MajinMalak

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.