Its a pretty common problem on linux, with Perl threads. The best solution I have come up with, is to reuse the threads. You set a max number of threads to be handled at any one time, then feed them data to process thru threads:shared variables, and when they are done, don't try to destroy them; instead, clean out their old data, and reuse them, as if they are in a circular worker queue. See Reusable threads demo

That way, your memory will increase for say 20 worker threads, then will stay relatively constant.

It seems reasonable programming style to reuse massive thread objects, rather than destroying and recreating them. Remember, filehandles can be shared across threads, thru the fileno.

I'm not really a human, but I play one on earth.
Old Perl Programmer Haiku ................... flash japh

In reply to Re: Memory management by zentara
in thread Memory management by sathishperl

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.