Unorthodox Embedding/XS madness!

Ouroborous has asked for the wisdom of the Perl Monks concerning the following question:

Okay... here's the basic task. I am writing an add-on for an existing program (which has its own scripting engine using its own "homegrown" scripting language that I'm not too fond of) to allow the existing program to call Perl scripts instead of "native" scripts.

I am nowhere near a Perl god -- I can write decent Perl code, but I've never before tried to do embedding or XSUB's. I've certainly learned a lot :)

First, an annoyance. I was getting MAJOR headaches when calling perl_parse. It seems that because I didn't have my scripts in the right path, the parse step was failing, and it was just returning an error code "9". Okay, I'm running in Win32, and in this OS, "9" doesn't mean anything that sounds even remotely like "file not found." I'd expect 2 or 0xC0000002 or something. So -- a way to find out what REALLY went wrong in embedding or xsubs would be fabulous. I downloaded the source code (for 5.8; I'm using ActiveState, which I believe is 5.6.1, another annoyance) but since I don't have debug symbols, I couldn't do anything truly useful with it. Any recommendations on how to get better error information?

Next... I'm probably going to be screamed at for this, but here goes... I need to inject callbacks (from Perl back to the embedding C++ app) into the Perl namespace. Now, I know I can write a .xs, and I've done this and it works, but because of how the original app works, it may be difficult to predict just exactly what the current directory is when I start the embedding. Meaning that if I don't have my .pm and .dll in exactly the right spot, the embedding fails (with some mysterious error like the beloved #9). Also, it makes it easier for me to distribute this if all of my Perl hooking modification is in ONE single file (an additional C module; of course the end user needs perl56.dll too).

SO, I poked around in perlembed, perlxs, and perlapi for a while, and figured out that I can inject a subroutine into Perl by calling um newXS. I hacked out all of the calls to dynaloader and exporter, then I hacked the code out of the .c file generated by MakeMaker and put it directly in my embedding app. Then I put in code to do a newXS and point to the XS code I'd inserted in the embedder.

Amazingly, this worked. Sort of... there were complications that only arose after the first eval_pv(). I think my problem was the order I did things in, and that I'm somehow messing up Perl's stack. That's just a hunch... The general sequence of operations was this...

1. "Main" app calls embedding app
2. Embedding app does perl_alloc(), perl_parse(), then "injects" the callback function using newXS.
3. perl_run() <-- sequence? what does perl_run() do, anyway?
4. eval_pv("MyInjectedXS::MyInjectedFunc(\"Perl now embedded!\")");

-- amazingly, this works, the "callback" function in my embedding app is called and I see what I expect to see... but now things get ugly...

5. embedder returns to "main" program after setting a flag that Perl is now initialized
6. "main" program eventually calls embedder shim for eval_pv()
7. embedder shim calls eval_pv()
8. <-- trap, every time

9. (theoretically, if we didn't trap) when "main" shuts down, it calls back into embedder
10. embedder does perl_destruct() and perl_free()

(NOTE: this may or may not be relevant; the embedded perl initialization and destruction, and the actual calls into the eval_pv() shim function, are taking place in two different threads...)
---

Obviously, my real problem here is that after the initial call, which works swimmingly, every eval_pv() traps. I can wrap it in a try/catch to avoid tanking the main program, but without debug symbols for Perl, I can't really figure out WHY it's trapping unless one of you worthy gentlemen (and ladies?) knows what's going on from looking at my sequence of operations.

I'm assuming that one of a few things is happening...

a. I'm trashing Perl's stack by calling an "injected" XSUB directly without doing some other kind of "glue code" that dynaloader is doing behind the scenes. Since I'm committed to using this method (direct injection of XSUB code into the perl namespace, rather than a full .xs, .pm, and .dll install), I'd like to know what the "glue code" is or at least where in the source to start looking.

b. My order of operations is wrong. I just totally guessed about WHEN I should "inject" my xsub code. Should I do it after perl_alloc() but before perl_parse()? Should I do it before or after perl_run()? What the heck does perl_run() do anyway -- do I need to even call it at all?

c. There's some other hidden factor that I don't know about. IE, running the perl_alloc, perl_parse, and initial eval_pv in one OS thread, and then running subsequent eval_pv's in another thread is a no-no or requires special consideration. Maybe there's some sort of cleanup/destructor activity that is taking place after my first "init" function is called and exits (preventing further eval_pv()'s or something) (EDIT: yes, I put the variable to hold the interpreter pointer into a static global variable; it's not local to the init function :) I don't know of any way of testing if my perl interpreter is still "valid" when I get called back into my eval_pv shim. Maybe I'm pulling a stupid no-brainer and doing something wrong that's totally unrelated to Perl (I don't think so -- I checked and rechecked my code like 20 times last night, but you never know).

So, ANY suggestions are welcome. Just please note that I'm not willing to go back to the method of using h2xs and makemaker and distributing a .pm and (additional) .dll for my final solution unless it is ABSOLUTELY not possible to inject xsubs dynamically.

Also, is there a better FM for the api (like what perl_run really does, or how newXS works behind the scened) than the perl source, so I can RTFM?

Thanks in advance, oh knowledgeable ones.

Edit by dws to fix title

Comment on Unorthodox Embedding/XS madness!

Replies are listed 'Best First'.
Re: Unorthodox Embedding/XS madness! by talexb (Chancellor) on Oct 10, 2002 at 02:29 UTC
This may not be a popular suggestion, but would it be possible to abandon XS and try Inline::C instead? I haven't used either, but Ingy's talk about how he came up with Inline::C at the Montreal YAPC was most illuminating. In addition, I see that CPAN also has Inline::CPR, something that lets you run Perl from within C. And in terms of having to inject subs dynamically, is that necessary? Could you instead subsitute a more general piece of code that Does The Right Thing when called? --t. alex but my friends call me T.	[reply]
Re: Re: Unorthodox Embedding/XS madness! by Ouroborous (Initiate) on Oct 10, 2002 at 06:46 UTC
I may have found the answer. My Perl distro (ActiveState, which is a 5.6.1 Win32 port), is compiled without either USE_THREADS or USE_ITHREADS. I'm not sure why that should matter (aren't these just for INTERNAL Perl threading?) but I've discovered that if I call eval_pv() from a __different O/S thread__ than the one I initialized the interpreter in, it traps. If I call from the same thread, it doesn't. I debugged it a little and found some calls to various TLS functions, so it appears that the simple expedient of storing my_perl in a global variable wasn't enough. Behind the scenes a pointer to it was getting stashed in thread-local storage, so that when I switched context over to my other thread, the pointer was null (different TLS instance) and we trapped as soon as we tried to use the interpreter pointer. VERY non-intuitive... As a C++ coder, I assume that if I dereference a global variable, it will have the same value over all functions in the module (assuming it isn't changed locally). However, this doesn't seem to match the actual behavior I'm seeing...	[reply]
Re: Unorthodox Embegging/XS madness! by Ouroborous (Initiate) on Oct 09, 2002 at 19:17 UTC
Oh yeah, that should be EmbeDDing, not EmbeGGing in the title. Maybe because I'm BEGGING for help, that's what made me typo it :)	[reply]
Re: Unorthodox Embedding/XS madness! by MZSanford (Curate) on Oct 10, 2002 at 13:50 UTC
I must say, i loved this post. This is how i learned embedding, and ran into many similar problems. The best companion i found is Extending and Embedding Perl (Manning). It covers alot of the small things that the perl docs don't. The only thing i can say, from my experiance, is that using Perl from diffrent threads (not creating threads with `use Threads`, but diffrent threads in an embedding application) is a big no-no. I have not worked with perl 5.8, but if you are running under 5.6.1, i think you will find it does not claim to be thread-safe, and, well, isnt :(. Hopefully someone else will post and say i am wrong, but i am afraid you may have to marshall all Perl calls into one thread and work from there. In a CB Style ... MZSanford Raises glass <MZSanford> Here's to hoping i am wrong from the frivolous to the serious	[reply] [d/l]
Re: Re: Unorthodox Embedding/XS madness! by Ouroborous (Initiate) on Oct 10, 2002 at 17:31 UTC
Thanks -- yeah, marshalling all the calls into one thread is what I had planned to do all along, ironically. This was just proof-of-concept code, so I hadn't bothered to put the call marshalling in place yet. I got thrown for a loop by the exception until I realized that the one thing I hadn't ruled out last night was threading issues. It was sort of a pain (for various reasons) to repro this in the actual embedded app, so I had created a "test app" that just embedded Perl the same way and called it directly. It didn't trap, which was frustrating. After I had my "threads epiphany", I added a couple lines of code to my test app to spawn the interpreter in another thread and hold it there until I signaled a shutdown. Then I called eval_pv() from the main thread and voila -- exception city. This was nice because the actual app was an injected DLL (long story) which made it VERY hard to debug the trap, whereas my test app was just a standalone executable, and it was a snap after the exception to step in and see where the interpreter was stashing the pointer to itself in thread-local storage. Oh well, live and learn. Well, it's good to know that it's not thread safe. That will hopefully keep me from shooting myself in the foot like this in the future. I can't wait until ActiveState gets their distro up to 5.8 -- I would REALLY like better built-in threading behavior (I'm assuming that since ithreads is included, the core Perl modules have been made thread-safe -- is this a reasonable assumption?) Thanks for the help and the pairs of eyes...	[reply]