A clean way to account for machine word sizes in XS code?

dec has asked for the wisdom of the Perl Monks concerning the following question:

Greetings, oh fellow monks! I've recently uploaded my first Perl module to CPAN (Math::FastGF2) and was very interested to see the automated test results on different platforms. But if you look there you'll see a lot of failed tests. I'm pretty sure that the problem with all the failed tests is that the test machines use different word sizes for C data types such as "unsigned short" and "unsigned long" and that's what's causing all the failures.

Recognising that this could have been an issue, I made sure to declare all the particular word sizes (1, 2 and 4-byte words) as typedefs in my C code, and made mention that these might need to be changed in the header files. However, as these are automated tests, nobody is looking at the header file to see what what might need to be changed in order to get the code to compile and run cleanly. So, what I'm looking for is an easy way to figure out which native C types have 8, 16 and 32 bits apiece at compilation time and to have my C/XS code use those data types. Can anyone help me?

There are a few more details I can add. First, I've noticed that if I define my XS prototypes using my user-defined data types (gf2_u8, gf2_u16 and gf2_u32) then the program won't compile correctly (I've tried using both typedefs and #defines). So at the moment I'm reduced to using my user-defined types for internal function calls and data structures, but having to use types like unsigned long for any function prototype that I want to be visible from Perl. Obviously this isn't ideal, as someone wanting to compile the code has to change all the typedefs and all the function prototypes that are meant to be visible to the Perl interpreter.

Second, I thought it should probably be possible to do this using typemaps, but my limited searching didn't turn up any information on doing something like I want.

The third and final bit of extra information is that I'm fairly used to using C-based configure scripts to figure out things like this and create header files based on the local configuration. I must admit that I'm only getting to grips with the MakeMaker system and the XS system in general, so I'm not even sure how to make my configuration script happen before the compilation proper. I hope you be mindful of my inexperience if you have a solution to offer.

Thank you, oh worthy ones!

Comment on A clean way to account for machine word sizes in XS code?

Replies are listed 'Best First'.
Re: A clean way to account for machine word sizes in XS code? by syphilis (Archbishop) on Aug 14, 2009 at 01:39 UTC
So, what I'm looking for is an easy way to figure out which native C types have 8, 16 and 32 bits apiece at compilation time and to have my C/XS code use those data types. Not sure that it's all that easy, but one way would be to have the Makefile.PL determine the sizes of the various types, and then use that info to construct the various typedefs. The ones that will interest you are: $Config{shortsize} $Config{intsize} $Config{longsize} $Config{ivtype} and $Config{ivsize} Not sure of how to determine charsize. The first 3 give you info about the sizes of the different types for the particular compiler, the last 2 give you the info about the size of the IV (the perl integer). Depending upon what those values are, you can have WriteMakefile() pass a define symbol to the build process (look for the DEFINE documentation in ExtUtils::MakeMaker), and have the header file write the appropriate typedefs based upon which symbol was defined: `#ifdef some_symbol // one set of typedefs #endif #if def another_symbol // another set of typedefs #endif . .` [download] I do pretty much this in Math::MPFR. I'm essentially looking only at $Config{nvsize} and $Config{ivsize} and I doubt that my approach is infallible, but it seems to work ok - afaik the failures I get there are for other reasons (most of which I don't really care about). Cheers, Rob	[reply] [d/l]
Re^2: A clean way to account for machine word sizes in XS code? by DrHyde (Prior) on Aug 14, 2009 at 09:41 UTC
char is 8 bits on every platform perl runs on. You could use the new C99 types, as that already has known-length types so you can declare a variable as, eg, a 16-bit int instead of just hoping that short is 16 bits. Unfortunately on some platforms (and possibly on some compilers on other platforms) you'll need to do some Magic to invoke the compiler in C99-mode. I suggest not bothering, until someone sends you a failure report. Then you ask them for a patch for their weird platform :-)	[reply]
Re: A clean way to account for machine word sizes in XS code? by tonyc (Friar) on Aug 14, 2009 at 01:50 UTC
For the typemap you need to add typemap entries that map your types to the same as the other int types, eg. in typemap: `gf2_u8 T_UV gf2_u16 T_UV gf2_u32 T_UV gf2_s8 T_IV gf2_s16 T_IV gf2_s32 T_IV` [download] ExtUtils::MakeMaker should pick up the type map automatically when you run Makefile.PL. As to whether those types are the correct types, one possibility is stdint.h, which requires a C99 compiler/libc, but in most cases, if the C compiler has 8 and 16 bit int types, char and short will be them. Unfortunately perl doesn't probe for stdint.h, so it's something you'd need to probe for yourself. For 32-bit, long might be 64-bit, so in the absence of inttypes.h you could use limits.h: `#include <limits.h> #if USHRT_MAX > 0x80000000 typedef unsigned short gf2_u32; #elif UINT_MAX > 0x80000000 typedef unsigned gf2_u32; #else typedef unsigned long gf2_u32; #endif` [download] According to the standard, they could all be 128-bit, but most likely char is 8 bit if the compiler has an 8-bit type, and unless the compiler has stdint.h, short will be the 16 bit type if there is one.	[reply] [d/l] [select]
Re: A clean way to account for machine word sizes in XS code? by syphilis (Archbishop) on Aug 14, 2009 at 09:37 UTC
I'm pretty sure that the problem with all the failed tests is that the test machines use different word sizes for C data types such as "unsigned short" and "unsigned long" and that's what's causing all the failures. I'm fairly sure you're on the right track there. I think you'll find that all of the failures have the following in common: $Config{intsize} is 4 $Config{longsize} is 8 $Config{ivtype} is long $Config{ivsize} is 8 (All of this can be gleaned from the tester's reports, btw.) The key thing might well be that a 'long' is 64 bits on these architectures. Cheers, Rob	[reply]
Success! (more words) by dec (Beadle) on Aug 14, 2009 at 22:35 UTC
After getting my other code out the door I came back to this problem and I think I have a solution that works. I basically took the typemap file mentioned by tonyc and the `$Config{foo}` method mentioned by syphilis. I only had one other problem relating to my top-level Makefile.PL not passing the -D defines through to the make in the subdirectory (I need them both for the library in the subdir and the Perl XS code in the toplevel that was going to link against it). I managed to solve this with a bit of custom make action: `sub MY::postamble { '$(MYEXTLIB): clib/Makefile cd clib && perl Makefile.PL DEFINE="' . (join ' ', @defines) . '" && $(MAKE) $(PASSTHRU) ...` [download] I also mentioned the possibility raised by DrHyde of using stdint.h in the comments in the library header file, and gave the user the option of using their own custom typedefs when calling the top-level Makefile.PL. All in all, thanks to your help it wasn't too difficult, and I think the code should be easy enough to understand too. Just what I wanted: simple and clean. Thanks again, dec	[reply] [d/l] [select]
Re: A clean way to account for machine word sizes in XS code? by dec (Beadle) on Aug 14, 2009 at 17:44 UTC
Thanks to all of you for your responses. They were very useful. Votes all round! I investigated some of the options but haven't implemented any of them yet. I'm inclined to try using the Config{"foo"} method first, along with a typemap file since it seems fairly simple. Anyway, I had to decide between fixing this and finishing the next release of the module. Getting the next release done won out this time, but I'll try to implement the fix for my 0.03 release. Thanks again, dec	[reply]