Can arbitrary XS/SWIG using modules be loaded into an embedded perl at runtime?

kingkongrevenge has asked for the wisdom of the Perl Monks concerning the following question:

I'm exploring ways to build a plugin system for an existing large C++ program. Embedding perl might be one good way to achieve this. The embedded perl could eval user code/modules to alter behavior.

However, it's absolutely critical that the user plugins can bring compiled code with them, i.e. code that is not already linked in the executable. Plugins would occasionally be doing some computationally intensive stuff internally where pure Perl wouldn't cut it. Is this possible?

My understanding is that to use object code from the perl hosting process you have to generate the xs_init code and link in the resulting object file. Is there some black magic way to support this by manually hacking the xs_init? No multi-process solutions are acceptable, though the perl hosting process is free to re-exec itself during some initialization stage; all the plugins to use would be known at start-up.

This might boil down to more of an operating systems question than a Perl question. Are there any ways to tell an executable to arbitrarily "load" an object file? DLL-Injection/LD_PRELOAD? I'm fairly ignorant of these things, if you can't tell.

Comment on Can arbitrary XS/SWIG using modules be loaded into an embedded perl at runtime?

Replies are listed 'Best First'.
Re: Can arbitrary XS/SWIG using modules be loaded into an embedded perl at runtime? by Joost (Canon) on Jul 23, 2008 at 23:25 UTC
I haven't written any embedded perl programs, but as far as I know, if you write your embedding code correctly, the perl code in question should be able to do anything that a "stand alone" perl program could. The setup code in C shouldn't take more than a dozen or so lines. As an aside, the reason I've never written a perl-embedding program, is that I've found it much to start with a perl program and load the C code using XS (IOW write your C code as a bunch of libraries and bootstrap it from perl (which could be as simple as doing one function call). update: just in case I wasn't clear: your embedded perl interpreter should be able to load compiled XS/C/C++ etc modules just like a normal perl interpreter can. "What should it profit a man, if he should win a flame war, yet lose his cool?"	[reply]
Re^2: Can arbitrary XS/SWIG using modules be loaded into an embedded perl at runtime? by kingkongrevenge (Scribe) on Jul 24, 2008 at 01:38 UTC
I will try it and report back here. If it works then the ExtUtils::Embed documentation is perhaps misleading, or I'm retarded. I interpreted the documentation to indicate that modules with binary components need explicit xs_init code. The alternate understanding is that the xs_init code is needed to call the modules from your C code. It's probably just the latter.	[reply]
Re^3: Can arbitrary XS/SWIG using modules be loaded into an embedded perl at runtime? by kingkongrevenge (Scribe) on Jul 24, 2008 at 04:32 UTC
I haven't tried it yet, but the "Pro Perl" book is a little more explicitly clear: If we want to use Perl modules that themselves make use of underlying C code (also called extensions), we have to go back and adjust the workings of our interpreter a little. In order to know how to load in the C library part of a Perl extension, the interpreter needs some additional help, which we provide by supplying a function pointer as the second argument to perl_parse. The function performs the job of bootstrapping the external C part of any extensions we want to use. Typically, we use the Dynaloader to dynamically load any modules on demand, so this is the only module we need to handle for most cases.	[reply]
Re: Can arbitrary XS/SWIG using modules be loaded into an embedded perl at runtime? by diotalevi (Canon) on Jul 24, 2008 at 04:37 UTC
It is always ok to load another module into a Perl runtime. Anytime you can run require(), you are able to load more perl and compiled C/XS/Swig code. Normal C and C++ modules are loaded by XSLoader and DynaLoader both of which come with perl. ⠤⠤ ⠙⠊⠕⠞⠁⠇⠑⠧⠊	[reply]