in reply to Re^5: perl 5.12.3 + threads + memory leak
in thread perl 5.12.3 + threads + memory leak

Sorry, but I don't think I am going to be able to help you.

Despite the amount of code you've posted, you've omitted a lot of the stuff that is likely to be the source of leaks.

Eg. the declarations (and the relative positioning in the source file(s?) of those declarations) for all the mysterious hashes that you use and maintain. If these hashes (eg.%db_connections, %db_ids_in_progress, %grand_hash, %running_threads ) are closed over by your thread procedure(s?), they are very likely sources of leaks, because:

Your code is full of anomalies and dubious coding practices:

  1. You use symbolic references:
    my ($cnf,$thread) = @_; if ($thread && !$db_connections{$cnf}{$thread}) { my $control = "dbh".$cnf.$thread; # ^^^^^^^^ $$control = DBI->connect( #^^^^^^^^ "dbi:Pg:database=dirotext;host=domain.com", 'user', 'pass' + ) or die $DBI::errstr;

    Which makes "settings stricts etc" dubious.

    The fact that you are including $thread into those symbolic names suggests that you are pooling dbi handles across threads which is a definite no-no.

  2. You add 'no parameters' prototypes to sub definitions that clearly take parameters:
    sub dbconn() { my ($cnf,$thread) = @_;

    And get away with it by overriding the prototypes by using &dbconn('pg1');

  3. You return a reference to a lexical hash, created in a thread, from that thread, and then dereference it to get at it's contents:
    my %response_hash = (); ... return \%response_hash; ... my %tr_res = %{$running_threads{$name}->join()};

    Whilst there is nothing wrong with that in as much as it works, the mechanics of what goes on under the covers to allow you to do it, make it unnecessarily prone to leakage.

    The reference you receive in the parent is (cannot be) the same reference that you take within the thread. In order to be able to give the parent thread a hash ref that it can dereference, the entire hash has first to be cloned into shared memory space.

    And then the first thing you do is clone it again into your local memory space.

    Far better to do:

    my %response_hash; ## No need to initialise lexicals. ... return %response_hash; ## Return a simple list of values ... my %tr_res = $running_threads{$name}->join(); ## Assign the list to a +local hash.

    One caveat of this is that you need to ensure that the thread procedure is given the a list context (so that it can return a list). That means that this (mess):

    $running_threads{$threadname} = threads->create(\&send_sms +_message, $db_ids_in_progress{$id}{'uid'}, $db_ids_in_progress{$id}{' +from'}, $db_ids_in_progress{$id}{'to'}, $db_ids_in_progress{$id}{'msg +'}, $id, $db_ids_in_progress{$id}{'dlr_track_id'}, $processing);

    Has to become:

    ( $running_threads{$threadname} ) = threads->create( \&send_sms_message, $db_ids_in_progress{$id}{'uid'}, $db_ids_in_progress{$id}{'from'}, $db_ids_in_progress{$id}{'to'}, $db_ids_in_progress{$id}{'msg'}, $id, $db_ids_in_progress{$id}{'dlr_track_id'}, $processing );

    Notice that the variable that will receive the thread handle is in parens, thus giving threads->create() a list context, which it duly passes on to the thread procedure.

The upshot is that the style of the code you've posted along with what you've chosen to omit, strongly suggest that you are one of those that eschew the use of strict (and that usually means you eschew warnings also; as confirmed by your earlier code). The fact that you feel the need to initialise hashes at various points indicates that these are almost certainly global variables with extended scopes.

Whilst you can get away with these archaic (perl4-ish) practices in single threaded code, mixing them with threads is a recipe for mysterious errors and leaks if not crashes.

Whilst it seems that threads currently leak on *nix, I'm guessing that the vast majority of your problems are self-inflicted. The bottom line is that unless you can post a full, strict & warnings clean, globals-free, version of your program that I can at least syntax check the results of changes using Perl, even if I cannot run it, wading through a mass of code like this and trying to guess where and how half the variables are declared is just too much work to consider.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^7: perl 5.12.3 + threads + memory leak
by kamenpetrov (Novice) on Feb 18, 2011 at 06:49 UTC
    Indeed, I have posted some not strict code. Sorry for that.

    I will try to rewrite everything using Thread::Pool.

    Thank you for taking time to provide that great answer !
      I will try to rewrite everything using Thread::Pool.

      That is a very bad idea.

        Why ? :)