Sorry, but I don't think I am going to be able to help you.

Despite the amount of code you've posted, you've omitted a lot of the stuff that is likely to be the source of leaks.

Eg. the declarations (and the relative positioning in the source file(s?) of those declarations) for all the mysterious hashes that you use and maintain. If these hashes (eg.%db_connections, %db_ids_in_progress, %grand_hash, %running_threads ) are closed over by your thread procedure(s?), they are very likely sources of leaks, because:

Your code is full of anomalies and dubious coding practices:

  1. You use symbolic references:
    my ($cnf,$thread) = @_; if ($thread && !$db_connections{$cnf}{$thread}) { my $control = "dbh".$cnf.$thread; # ^^^^^^^^ $$control = DBI->connect( #^^^^^^^^ "dbi:Pg:database=dirotext;host=domain.com", 'user', 'pass' + ) or die $DBI::errstr;

    Which makes "settings stricts etc" dubious.

    The fact that you are including $thread into those symbolic names suggests that you are pooling dbi handles across threads which is a definite no-no.

  2. You add 'no parameters' prototypes to sub definitions that clearly take parameters:
    sub dbconn() { my ($cnf,$thread) = @_;

    And get away with it by overriding the prototypes by using &dbconn('pg1');

  3. You return a reference to a lexical hash, created in a thread, from that thread, and then dereference it to get at it's contents:
    my %response_hash = (); ... return \%response_hash; ... my %tr_res = %{$running_threads{$name}->join()};

    Whilst there is nothing wrong with that in as much as it works, the mechanics of what goes on under the covers to allow you to do it, make it unnecessarily prone to leakage.

    The reference you receive in the parent is (cannot be) the same reference that you take within the thread. In order to be able to give the parent thread a hash ref that it can dereference, the entire hash has first to be cloned into shared memory space.

    And then the first thing you do is clone it again into your local memory space.

    Far better to do:

    my %response_hash; ## No need to initialise lexicals. ... return %response_hash; ## Return a simple list of values ... my %tr_res = $running_threads{$name}->join(); ## Assign the list to a +local hash.

    One caveat of this is that you need to ensure that the thread procedure is given the a list context (so that it can return a list). That means that this (mess):

    $running_threads{$threadname} = threads->create(\&send_sms +_message, $db_ids_in_progress{$id}{'uid'}, $db_ids_in_progress{$id}{' +from'}, $db_ids_in_progress{$id}{'to'}, $db_ids_in_progress{$id}{'msg +'}, $id, $db_ids_in_progress{$id}{'dlr_track_id'}, $processing);

    Has to become:

    ( $running_threads{$threadname} ) = threads->create( \&send_sms_message, $db_ids_in_progress{$id}{'uid'}, $db_ids_in_progress{$id}{'from'}, $db_ids_in_progress{$id}{'to'}, $db_ids_in_progress{$id}{'msg'}, $id, $db_ids_in_progress{$id}{'dlr_track_id'}, $processing );

    Notice that the variable that will receive the thread handle is in parens, thus giving threads->create() a list context, which it duly passes on to the thread procedure.

The upshot is that the style of the code you've posted along with what you've chosen to omit, strongly suggest that you are one of those that eschew the use of strict (and that usually means you eschew warnings also; as confirmed by your earlier code). The fact that you feel the need to initialise hashes at various points indicates that these are almost certainly global variables with extended scopes.

Whilst you can get away with these archaic (perl4-ish) practices in single threaded code, mixing them with threads is a recipe for mysterious errors and leaks if not crashes.

Whilst it seems that threads currently leak on *nix, I'm guessing that the vast majority of your problems are self-inflicted. The bottom line is that unless you can post a full, strict & warnings clean, globals-free, version of your program that I can at least syntax check the results of changes using Perl, even if I cannot run it, wading through a mass of code like this and trying to guess where and how half the variables are declared is just too much work to consider.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re^6: perl 5.12.3 + threads + memory leak by BrowserUk
in thread perl 5.12.3 + threads + memory leak by kamenpetrov

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.