Okay... here's the basic task. I am writing an add-on for an existing program (which has its own scripting engine using its own "homegrown" scripting language that I'm not too fond of) to allow the existing program to call Perl scripts instead of "native" scripts.

I am nowhere near a Perl god -- I can write decent Perl code, but I've never before tried to do embedding or XSUB's. I've certainly learned a lot :)

First, an annoyance. I was getting MAJOR headaches when calling perl_parse. It seems that because I didn't have my scripts in the right path, the parse step was failing, and it was just returning an error code "9". Okay, I'm running in Win32, and in this OS, "9" doesn't mean anything that sounds even remotely like "file not found." I'd expect 2 or 0xC0000002 or something. So -- a way to find out what REALLY went wrong in embedding or xsubs would be fabulous. I downloaded the source code (for 5.8; I'm using ActiveState, which I believe is 5.6.1, another annoyance) but since I don't have debug symbols, I couldn't do anything truly useful with it. Any recommendations on how to get better error information?

Next... I'm probably going to be screamed at for this, but here goes... I need to inject callbacks (from Perl back to the embedding C++ app) into the Perl namespace. Now, I know I can write a .xs, and I've done this and it works, but because of how the original app works, it may be difficult to predict just exactly what the current directory is when I start the embedding. Meaning that if I don't have my .pm and .dll in exactly the right spot, the embedding fails (with some mysterious error like the beloved #9). Also, it makes it easier for me to distribute this if all of my Perl hooking modification is in ONE single file (an additional C module; of course the end user needs perl56.dll too).

SO, I poked around in perlembed, perlxs, and perlapi for a while, and figured out that I can inject a subroutine into Perl by calling um newXS. I hacked out all of the calls to dynaloader and exporter, then I hacked the code out of the .c file generated by MakeMaker and put it directly in my embedding app. Then I put in code to do a newXS and point to the XS code I'd inserted in the embedder.

Amazingly, this worked. Sort of... there were complications that only arose after the first eval_pv(). I think my problem was the order I did things in, and that I'm somehow messing up Perl's stack. That's just a hunch... The general sequence of operations was this...

1. "Main" app calls embedding app
2. Embedding app does perl_alloc(), perl_parse(), then "injects" the callback function using newXS.
3. perl_run() <-- sequence? what does perl_run() do, anyway?
4. eval_pv("MyInjectedXS::MyInjectedFunc(\"Perl now embedded!\")");

-- amazingly, this works, the "callback" function in my embedding app is called and I see what I expect to see... but now things get ugly...

5. embedder returns to "main" program after setting a flag that Perl is now initialized
6. "main" program eventually calls embedder shim for eval_pv()
7. embedder shim calls eval_pv()
8. <-- trap, every time

9. (theoretically, if we didn't trap) when "main" shuts down, it calls back into embedder
10. embedder does perl_destruct() and perl_free()

(NOTE: this may or may not be relevant; the embedded perl initialization and destruction, and the actual calls into the eval_pv() shim function, are taking place in two different threads...)
---

Obviously, my real problem here is that after the initial call, which works swimmingly, every eval_pv() traps. I can wrap it in a try/catch to avoid tanking the main program, but without debug symbols for Perl, I can't really figure out WHY it's trapping unless one of you worthy gentlemen (and ladies?) knows what's going on from looking at my sequence of operations.

I'm assuming that one of a few things is happening...

a. I'm trashing Perl's stack by calling an "injected" XSUB directly without doing some other kind of "glue code" that dynaloader is doing behind the scenes. Since I'm committed to using this method (direct injection of XSUB code into the perl namespace, rather than a full .xs, .pm, and .dll install), I'd like to know what the "glue code" is or at least where in the source to start looking.

b. My order of operations is wrong. I just totally guessed about WHEN I should "inject" my xsub code. Should I do it after perl_alloc() but before perl_parse()? Should I do it before or after perl_run()? What the heck does perl_run() do anyway -- do I need to even call it at all?

c. There's some other hidden factor that I don't know about. IE, running the perl_alloc, perl_parse, and initial eval_pv in one OS thread, and then running subsequent eval_pv's in another thread is a no-no or requires special consideration. Maybe there's some sort of cleanup/destructor activity that is taking place after my first "init" function is called and exits (preventing further eval_pv()'s or something) (EDIT: yes, I put the variable to hold the interpreter pointer into a static global variable; it's not local to the init function :) I don't know of any way of testing if my perl interpreter is still "valid" when I get called back into my eval_pv shim. Maybe I'm pulling a stupid no-brainer and doing something wrong that's totally unrelated to Perl (I don't think so -- I checked and rechecked my code like 20 times last night, but you never know).

So, ANY suggestions are welcome. Just please note that I'm not willing to go back to the method of using h2xs and makemaker and distributing a .pm and (additional) .dll for my final solution unless it is ABSOLUTELY not possible to inject xsubs dynamically.

Also, is there a better FM for the api (like what perl_run really does, or how newXS works behind the scened) than the perl source, so I can RTFM?

Thanks in advance, oh knowledgeable ones.

Edit by dws to fix title

In reply to Unorthodox Embedding/XS madness! by Ouroborous

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.