however it seems that the memory being used to hold this file is not being released, and I get huge build-ups of wasted memory.

You have two problems. The first is a misunderstanding about how Perl uses memory. When you do something that requires more memory than Perl has available, Perl grows its internal memory pool by requesting more memory from the operating system. Perl then allocates internally from this pool. As far as I know, there's no provision at present for returning memory to the operating system. A common trick in long-running Perl applications is to save state in a file, then have the application re-exec yourself.

The second problem is that you're using "slurp mode" to read the entire file at once. Unless you have a search pattern that extends across multiple lines, you can read, and search, the file line-by-line instead. The additional work this entails might be offset by the lower memory footprint it requries.

Update: If your search patterns does span lines, you might consider the technique described in Matching in huge files.


In reply to Re: Where's the leak? by dws
in thread Where's the leak? by tekkie

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.