How to author a module with optional XS

NERDVANA has asked for the wisdom of the Perl Monks concerning the following question:

The usual pattern for optional-XS modules is to have one dist that is the pure-perl implementation and another dist that overrides the first with some XS speedups.

This works, but often people list the pure-perl module as a dependency and then sometimes the XS one doesn't get installed for the average end-user, letting them suffer with low performance not realizing they could speed it up with an additional module.

Has anyone explored the possibility of packaging the XS and PP together in the same dist so that the XS is compiled and installed iff a C compiler is available? Or even better, skips installing the XS if it fails to compile and install for any reason?

I was toying with the idea, and thought a quick way to get there without a bunch of Makefile.pm shenanigans would be if I create a normal XS dist and then include that as a subdirectory of the PP dist. Then I just start a subshell to run the nomal make process in that subdirectory, and if it fails, I ignore it and continue installing and testing the PP module.

Does anyone see any problems with that? Had this been solved better ways by other modules I could look at as an example?

Comment on How to author a module with optional XS

Replies are listed 'Best First'.
Re: How to author a module with optional XS by hv (Prior) on Aug 30, 2023 at 20:47 UTC
Math::Prime::Util has pure-perl and XS implementations of everything, with options to override which to use via the environment variable `MPU_NO_XS`, and specific documentation on some methods to warn that they will be unusually slow in pure perl. It's a pretty mature and successful distribution, so one worth looking at (though it hasn't had a new release for a while now - I remain hopeful that there will be one soon). I don't believe it attempts to automate the choice however: as far as I recall, the user decides whether to build the XS, and whether to use it in any given instance. That feels to me like a better approach than having the build process decide for itself - I wouldn't want a module to silently install the pure-perl code and claim success just because it tripped over some trivial, easily-fixed problem building the XS code.	[reply] [d/l]
Re^2: How to author a module with optional XS by NERDVANA (Priest) on Aug 30, 2023 at 23:30 UTC
I wouldn't want a module to silently install the pure-perl code and claim success just because it tripped over some trivial, easily-fixed problem building the XS code Well, that's a good point. Trying to automatically make it fast could just as easily fail and make it silently not-fast. What about trying to detect presence of a functioning C compiler during Makefile.PL and then spitting out a dependency on the ::XS variant? Something like: If an environment variable says not to try XS, proceed with normal make Else, look for a C compiler and relevant tooling. If found, end the Makefile.PL with a dependency on MyModule::XS cpanm switches to installing MyModule::XS cpanm comes back and finishes installing MyModule. Are there official methods to detect whether XS is supported by a Perl environment? I thought maybe I could find out whether Devel::CheckLib finds a compiler, but it appears that Devel::CheckLib probably can't be installed unless it does find a compiler. So I can't list Devel::CheckLib as a dependency for a module that I want to be installable on compiler-less perls. Maybe I shouldn't worry about compiler-less perl environments? (are there any where a compiler can't be installed?) and just focus on making sure my module can work properly with fat-packing, since I think that is the main reason that people try to avoid XS dependencies?	[reply]
Re^3: How to author a module with optional XS by hv (Prior) on Aug 30, 2023 at 23:44 UTC
I don't know what capabilities there are to detect whether C compilation is available. Generally, however, the user will know - and might want to skip the C compilation even when a compiler is available, for whatever reason. So my inclination would be to let the user choose, and to concentrate on providing good error messages if you can't do what they ask you to (and at most a warning if you think what they've asked for isn't the best option for them). For what it's worth, one of my recent maths projects (written in C) is being run by a number of people on Windows, but despite several of them being programmers only one of them was prepared to actually compile the code (who then provided a binary I could distribute to the others) - even while some of them were asking for features that were already available as compile-time options. I do not pretend to understand this, but I think I can reasonably conclude that some people have a complicated relationship with the concept of a compiler.	[reply]
Re^4: How to author a module with optional XS by afoken (Chancellor) on Aug 31, 2023 at 13:57 UTC
Re^5: How to author a module with optional XS by NERDVANA (Priest) on Sep 02, 2023 at 07:01 UTC
Re^5: How to author a module with optional XS by etj (Priest) on Mar 02, 2025 at 12:15 UTC
Re^3: How to author a module with optional XS by Anonymous Monk on Sep 01, 2023 at 05:02 UTC
Re "are there any where a compiler can't be installed?" - answer - Production systems usually ban compilers for security.	[reply]
Re^4: How to author a module with optional XS by Anonymous Monk on Sep 01, 2023 at 09:58 UTC
Re: How to author a module with optional XS by hippo (Archbishop) on Sep 02, 2023 at 09:36 UTC
The Lancaster Consensus says: Some distributions offer an "XS" version or a "Pure Perl" version that can be selected during configuration. Currently, each of these has their own way for users to indicate this, which makes it impossible for CPAN clients or other build tools to help users select automatically. Going forward, the "spec" for Makefile.PL and Build.PL will include command line options to request a "pure Perl only" build. These will be: PUREPERL_ONLY=1 (for Makefile.PL) --pureperl-only (for Build.PL) So, applying the inverse logic, if the user or their environment has not specifically requested the pure-perl version by this mechanism then you would be abiding by the consensus to attempt the XS install and inform the user of failure to do so when appropriate. That would probably be my approach but it's hypothetical as I don't maintain any XS modules. 🦛	[reply]
Re^2: How to author a module with optional XS by NERDVANA (Priest) on Sep 02, 2023 at 10:10 UTC
Ha-ha! This is what I was looking for. So I can just start the Makefile.PL with a check for an environment variable that disables all the XS, and document the module as supporting this feature. And if I go searching this keyword maybe I'll find existing tooling for it.	[reply]
Re: How to author a module with optional XS by tobyink (Canon) on Aug 31, 2023 at 11:49 UTC
Something nice might be if somebody released a module called `issing::XS` on CPAN, so that: `perl -Missing::XS your/script.pl` [download] Would detect, for example, if JSON.pm got loaded but JSON::XS wasn't installed and print out a warning. Obviously it would need to have a hard-coded list of modules that have XS backends, so would need to be updated occasionally. (It could also perhaps check and warn if issing::XS itself were out of date!) Hire me at Toby Ink Ltd or Join my OnlyFans	[reply] [d/l] [select]