in reply to Re^10: Perl crash during perl_clone
in thread Perl crash during perl_clone

Your code, with a couple of minor corrections:

  1. Two cases of adding 1 or 2 to the C variable name: call_sv( callback1, G_DISCARD );
  2. Six cases of adding the $ sigil to Perl variable names: my $cb1_cnt = 0;

runs perfectly here! (Not bad for code you couldn't test :). So, where does that leave us?

However, my confusion was why there would be 'multiple' contexts: without any perl_clone, I expected only one perl context now.

If there is only one Perl thread, there will be only one Perl context.

The problem is, when the callback is triggered and you enter the C-level callback routine (CCB) ready to call the Perl-level callback (PCB), Perl requires that context be established (set), *before* you call the PCB. But, as the CCB is just a C subroutine, that context is not established for you, as it would be with an XS routine. So, you have to do it yourself.

Hence the need to hive it off somewhere, and establish it before calling back into Perl. "it" here being the global PerlInterpreter *saved(1|2) contexts.

The local PerlInterpreter *temp contexts within the CCBs are actually redundant in my example code, because I know that the callbacks will only ever be initiated from the C-level timer threads. Ie. they will never be called from an existing Perl context.

But, and it is a big one, you haven't clarified--nor from memory, even mentioned--what triggers the callbacks in your real application? And this could be very significant.

For example, on my OS (windows), I know that asynchronous IO callbacks generally run in the context of a completely new thread, started & 'injected' into the Perl process, by the OS. As such, they are clean environments in which we can pretty much do what we like, because as soon as we return from them, they will just disappear with no lasting effects beyond what we program into them.

However, if you are running on a *nix system, asynchronous events can be triggered through signals. Which means that the CCB will be invoked not in a new, clean thread, but rather in an existing (Perl created) thread, via a longjump. Which means that you need to be very careful what you do to both the global state, and thread-local state of perl during those CCBs. Hence the local save/restore of the temp perl context is my attempt to guess what might be necessary on your platform. (The errors you mention tell me I got it wrong!).

And this is where my ability to help you curtails rapidly.

Even if you can provide a clear and concise description of what triggers these callbacks, the chances are I will have little or no experience of those mechanisms on your platform. And even if you could put together some concise, stand-alone code that compiles on your platform and demonstrates the problems you are encountering, it almost certainly wouldn't compile and run here. Which again would leave me in the position of trying to run the code in my head--for an OS I have little knowledge of.

The bottom line is that without a concise, easily buildable, runnable demonstration of the problem, there is little anyone can really do to help you.

With such a demo, then you may well find a willing, thread-aware user on your platform (or one sufficiently similar) that can take up the baton. Either here, or perhaps the appropriate CPAN forum or mailing list (See the bottom of the threads POD).

Such a demo might even draw a response from the most-likely-to-be-able-to-help people, those on p5p. But you need to make it easy for them to try your code and see the problem.

I'm not abandoning you, just keeping it realistic about my chances of being able to help you.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP an inspiration; A true Folk's Guy

Replies are listed 'Best First'.
Re^12: Perl crash during perl_clone
by perlmonk1729 (Acolyte) on Oct 29, 2010 at 18:56 UTC
    runs perfectly here!

    You must be kidding..Holy cow! Indeed, where does it leave us know..

    Guess, it could be platform-dependent stuff. I'll definitely need to get a demonstration of the problem first. I'd have to learn Inline to get rid of the compile errors (about "CALLBACK" and also that "thread(1|2) are undefined)...or revise the example from my earlier posts. I may pick the latter approach.

    I completely understand your "warnings" on how much you could help, but I feel you will still be quite helpful. So, let me describe a bit more of the project, anyhow.

    We are on *nix platforms (ubuntu and fedora mainly but RH too). The 'arch' is client-server : a standalone executable server (all C). A client-lib (all C) that I've wrapped with SWIG. An application (perl script, in our case) uses the clientlib API to exercise server functionality. Client-server communicate via sockets. Client API spawns threads and sends msgs to server to 'do things'. Server program will use some HW to do its work. When done, server sendsback msgs, results via the socket. The threads will then call the approriate CCBs based on various msgs received from server. Make sense? I hope :-)

    Which means that the CCB will be invoked not in a new, clean thread, but rather in an existing (Perl created) thread, via a longjump

    As in the sample code we exchanged, in my real project too, the threads are created on the C side and not by Perl. So, it seems this italisized statement would not apply, right?

[reply]
      but I feel you will still be quite helpful. So, let me describe a bit more of the project

      Then I'm going to need much more detail about the client API and the perl script that is using it.

      You'd need to considerably expand these two steps of the description:

      Client API spawns threads and sends msgs to server to 'do things'.

      The threads will then call the approriate CCBs based on various msgs received from server.

      Questions that come to mind:

      • One (C) thread per server connection?
      • Does that thread procedure sit in a block loop?
      • One PCB per thread?
      • What does the Perl script do whilst waiting for PCBs to be called?
      • Is there only (ever?) one Perl thread?
      • Can two C-threads attempt to call the same PCB? Concurrently?
      in my real project too, the threads are created on the C side and not by Perl. So, it seems this italisized statement would not apply, right?

      If there is only (ever?) one Perl thread in the client program, then there will only be one Perl context.

      But, in that case, by using dTHX: within the CCBs prior to calling the PCBs, all the context issues should (I believe) be resolved. That is certainly the case in both my test code, and my corrected version of your updates to that code, on my OS.

      The fact that your real code is failing, and your attempts to use my test code fail at compile-time, strongly suggests that your problems distinctly fall into the realms of "platform dependant". Hence, my strong reservations about my ability to help you further.

      I could continue to try and re-create your circumstances based entirely upon your (to date, scant) descriptions, but if the underlying problem in your real code would not manifest itself on my platform, I am never going to be able to re-create your problems here. No matter how much effort I expend.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
        I'm traveling and will be limited access to work on the code. I'll respond to your questions now, but would suggest that you do not spend any time besides 'thinking' and asking me further questions/or offer 'gut-feel' ideas.

        One (C) thread per server connection?

        Yes. Note: Several of the C API can end-up creating independent C threads, each with a separate connection to the server.

        Does that thread procedure sit in a block loop?

        Yes. Depnding on 'responses' from the server, it will take appropriate actions, including invoking CCB at appropriate times.

        One PCB per thread?

        Not necessarily.

        What does the Perl script do whilst waiting for PCBs to be called?

        Not much, Id say. The script typically goes like this : Call an API, go into a "while" loop, waiting on a flag for the expected PCB to complete. PCB sets the flag, passes the results back thru another variables. (Perl HASH wrappers for the underlying C structures). Depending on the results, proceeds with the next action/API until a pass/fail verdict can be produced.

        Is there only (ever?) one Perl thread?

        There are no perl_clone's anymore. So, only one Perl context created by perl.exe itself. But, you probably didnt mean that by Perl thread: I do not create any perl level threads (itheads?) in the perl script. Its a 'main' & a bunch of helper subs (in a separate .pm) and a bunch of PCBs.

        However, I would like, if possible, to allow script developer flexibility to use perl threads in the future.

        Can two C-threads attempt to call the same PCB?

        No, though there is a many-to-1 relation between PCBs and C-threads, each PCB is handled only by one C-thread.

        Concurrently?

        Umm..technically it maybe possible : I mean, two C-threads each calling a separate PCB 'concurrently' (i.e a 2nd PCB is invoked before 1st PCB completes). I believe all the C-threads are at the same priority, so its not likely. However, I'm certain there is no such situation occuring with the simple scripts that are giving me problems now.

        The system follows an multi-threaded, multi-process, async event-driven programming model. The Perl script built using the wrapped C-API is the driver/initiator of all activities. Each perl script would try to exercise a specific scenario that that could occur in a production app.

        I am more than happy to provide further details, but may not be seeing what details will help you. So, appreciate you asking specific question.

        Hope this helps and thanks once again!

[reply]

In Section Seekers of Perl Wisdom