in reply to Re^4: threads: work crew memory leak
in thread threads: work crew memory leak

Creating a timer thread to kill timed out threads seemed very easy and handy.

That sounds simple and convenient, but threads have state. And if you go around just killing them, you don't give them the opportunity to clean up after themselves and that's where 'mysterious' memory leaks arise.

Creating a fork within a thread with a signal handler killing the fork on the other hand would be a lot more expensive.

I certainly agree with you that that is not a viable option.

But there are better alternatives to both those approaches. The details depend upon the type of "network related issues" you are concerned about?

In general, using cancellable asynchronous IO or non-blocking IO, within the thread, to allow you to abandon slow or broken connections, is a viable alternative. It allows you to retain the benefits of threading: simplicity of 'serial flow' architecture; scalability across cores; whilst avoiding the traps of 'throwing threads at the problem' & the difficulties of clean-up associated with abandoning threads.

You gain the benefits of a non-blocking IO, without the need to invert the flow of control, or have the need for abandonable IO to dictate the architecture of your entire application.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP an inspiration; A true Folk's Guy

Replies are listed 'Best First'.
Re^6: threads: work crew memory leak
by rakzer (Novice) on Oct 18, 2010 at 13:30 UTC
    That sounds simple and convenient, but threads have state. And if you go around just killing them, you don't give them the opportunity to clean up after themselves and that's where 'mysterious' memory leaks arise.
    What would be the right way(tm) to kill a hanging thread as far as proper garbage collection goes?
    But there are better alternatives to both those approaches. The details depend upon the type of "network related issues" you are concerned about?
    Basically all the program does is retrieving the RFC 2616 status code of a large number (+500k) of distfiles from various different http/ftp servers using HTTP::Request and LWP::UserAgent.

    Setting $ua->timeout(10); deosn't seem to be working at all. I'm now doing:

    my $code;
eval {
    local $SIG{ALRM} = sub {die "alarm\n"};
    alarm $defcfg->{'link_check_timeout'};

    # LWP stuff
    $code= ...;

    alarm 0;
}

    [download]
    Not sure if that's the best thing but it seems to work better than setting an LWP timeout.
[reply]
[d/l]
[select]
      What would be the right way(tm) to kill a hanging thread as far as proper garbage collection goes?

      There is no right way to kill a thread.

      Even at the OS level, the TerminateThread() api--if your OS has one--is designated an action of extreme last resort:

      Remarks

      TerminateThread is used to cause a thread to exit. When this occurs, the target thread has no chance to execute any user-mode code. DLLs attached to the thread are not notified that the thread is terminating. The system frees the thread's initial stack.

      Windows Server 2003 and Windows XP/2000: The target thread's initial stack is not freed, causing a resource leak.

      TerminateThread is a dangerous function that should only be used in the most extreme cases. You should call TerminateThread only if you know exactly what the target thread is doing, and you control all of the code that the target thread could possibly be running at the time of the termination.

      And that's from C. From Perl, you simply have no way of meeting those requirements.

      Not sure if that's the best thing but it seems to work better than setting an LWP timeout.

      The LWP timeout is a generic setting that it tries to apply across a wide range of protocols and phases of the communication process. But during socket operations there are many phases. The initial connection; the ack/nak responses; the transmission of blocks of content and the associated handshaking; the shutdown etc.

      If, for example, you are talking to a slow or busy server, it is quite possible for each phase to take just less than the timeout value you set. With the result that the overall time for the communication can be several time longer than the timeout value. Hence, you are left wondering why did it continue for 30 or 40 seconds when I set a 10 seconds timeout? And the answer is because no individual stage of the communication broached the 10 second limit.

      If your use of alarm is meeting your requirements, then it is as good as any other way. I've used it myself successfully, but I have found that there appear to be some points in the process of communicating with a server that it won't successfully interrupt due to "safe signals".

      If you find that you sometimes get servers that cause your threads to hang around blocked for longer than is desirable, despite your use of alarm, then you might want to try adding Perl::Unsafe::Signals to your eval block. See the module's synopsis for how.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
        Thank you for all the professional advise.
[reply]

In Section Seekers of Perl Wisdom