in reply to Perl crash during perl_clone

My post on perl5-porters didnt get any reponses.

If you supplied a similarly minimal amount of information there as here, that is not surprising. It will be very difficult to reach any definitive conclusions based up such scant information.

As I said in a previous thread, if you don't post representative, working code to demonstrate your problem, then anyone attempting to help you has to try and re-create your scenario from your very brief description. And that really is asking far too much of volunteers.

I notice that you actually did post some code in a previous thread, but you did so as replies to yourself, which means that no one else ever received a notification of those posts.

Ie. It is probable that none of the monks involved in that thread, including cdarke who actually attempted to reproduce your failing scenario, almut nor I were even aware that you had posted it. I certainly wasn't.

Attempting to read between the lines of the information you have provided here (and in your related previous threads), you are attempting to call a coderef (callback)--in an interpreter/thread that you spawned yourself from within C--that was passed into your C code from another interpreter/thread spawned by Perl.

With the best will in the world, that is never going to work reliably. I'd be amazed if it ever worked at all. Though from your description it appears it has and does--so color me amazed.

The only way I can see it working sometimes and not others, is if your code is dependant upon which thread happens to be running when the callback is made. That is to say, if at the instant the asynchronous event (socket read completion?) that triggers the callback, the same thread that passed the callback coderef to your C code happens to be running, then it will likely work. If however, the other thread happens to be running, then it doesn't.

If this is the case--which should be fairly easy to verify by having your C/XS code log the current thread id: a) when it is passed the callback coderef; b) just before it invokes that callback. And if it is the case, the resolution would be to ensure that when your C-code creates it's own interpreter/thread, it follows the exact same steps as used by threads::create(). Of course, in reality, it would almost certainly be easier to allow Perl to that for you.

But then comes the question of why you feel the need to create a new interpreter in the first place? And for anyone to be able to understand that, you will have to describe the overall functionality and methodology of your complete application in far more detail than you have to date.

But...there is a simpler route. Take your 10MB of C code out of the equation. Once you have a reliable demonstration of passing in a Perl coderef to C code, which is subsequently called back at a later time--possibly triggered by a timer--in a multi-threaded environment, it should then be pretty easy to port that back to your real code.

Perhaps based on the code you posted in that other thread, create a minimal, runnable example of the callback scenario you require. But do it in a single post--as a reply to someone other than yourself.

Preferably do it as a single file using Inline::C to make it easy for others to download and run. I for one wouldn't know what to do with all the separate files you posted in that other thread, in order to get to the point where I could actually run the code.

If you cannot see how to do it as a single file using Inline::C, then at least wrap the multiple files up into a proper CPAN -style distribution and make it available somewhere, so that others can use CPAN to build and test it.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP an inspiration; A true Folk's Guy

Replies are listed 'Best First'.
Re^2: Perl crash during perl_clone
by perlmonk1729 (Acolyte) on Oct 27, 2010 at 11:21 UTC
    Hi BrowserUk:

    Yes, I meant to put a link to the old post but missed it. Thx for finding it yourself! That code is still quite valid. I also did not realize that others did not get notification of my replies.

    Reg "you are attempting to call a coderef (callback)--in an interpreter/thread that you spawned yourself from within C--that was passed into your C code from another interpreter/thread spawned by Perl."

    I think you got the gist of my approach, except that the the coderef was passed into C code from Perl's "default" interpreter (the instance created automatically when I run perl.exe) & NOT from another interpreter/thread spawned by Perl (not sure the difference matters).

    I found this approach from perlembed, other posts and online references. It seems others had it working similarly.

    I'll try to package the sample code as downloadable. However, CPAN packaging is new to me so need to research it.

    When this problem occurs, there are only two "threads" of execution..perl.exe's interpreter and the 2nd interpreter/thread I creaetd. Anyway, I'm open to trying any hunches..so, will try this as well.

    I wondered if the crash could be due to perl's "default" interpreter changing data structures while the cloning (perl_clone) is still executing. I was planning to explore 'blocking' the default interpreter until the perl_clone is complete.

    Appreciate your time/comments very much!

[reply]
      except that the the coderef was passed into C code from Perl's "default" interpreter (the instance created automatically when I run perl.exe) & NOT from another interpreter/thread spawned by Perl (not sure the difference matters).

      What you've described is exactly what I was trying and failing to describe. Two interpreters, one created by perl, one by your C code. And passing a code reference between those two. Perl will not allow you to attempt to pass a coderef between interpreters:

      perl -Mthreads -Mthreads::shared -Mstrict 
-wle"
    my $c:shared; 
    async{ 
        ## This next line triggers the "Invalid value for shared scala
+r"
        $c = sub{ print 'Hi'}; 
        sleep 100
    }->detach;; 
    sleep 3;
    $c->();
    sleep 100"

Thread 1 terminated abnormally: Invalid value for shared scalar at -e 
+line 1.
Can't use an undefined value as a subroutine reference at -e line 1.

      [download]

      But, by passing the coderef via C, you are by-passing this protection. Hence, as I surmised, if the callback gets triggered at a point in teh application when the "wrong" interpreter has the cpu, then it fails. Whereas if the originating inpterpreter is in control, it works,

      The solution I'm afraid is to re-think your methodology. If you can describe why you are starting a second interpreter, we might be able to see a solution.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
        if the callback gets triggered at a point in teh application when the "wrong" interpreter has the cpu, then it fails

        There are only two threads/interpreters and the thread that cloned an interpter does nothing else but wait for the right event to invoke the callback. So, bit confused what could be the 'wrong' interprter

        The solution I'm afraid is to re-think your methodology. If you can describe why you are starting a second interpreter, we might be able to see a solution.

        Infact, I did not know anything about interpters or cloning when I started this project. I learnt how to modify the SWIG generated wrapper to register the callback (RegisterCB in my example): I cache the perl sub reference and instead register a proxy C method (InvokeCB) into the C-api interface. When an unrelated thread (BGThread) called InvokeCB, it sets up the call-stack properly to call perl subroutines and then calls the sub via the cached reference.

        However, the "call_sv" would always crashed - 100%. Then I stumbled across the perl_clone() and found many posts that claimed it was required because BGThread cannot call code into an interpreter it did not create (i.e perl.exe).

        Having a cloned interpter allowed call_sv to work reliably..until my latest problems. You can see this link about my previous experiences that got me here. http://markmail.org/thread/cjbhybjfikuirvud

        ps: I'm in asian timezone, so responses may seem to have a 'lag'.
[reply]

In Section Seekers of Perl Wisdom