Note: If you don't get the gag, try googling for 'Martin Luther' and '95 theses' or look here.

This really derives from increasing frustration with XS & C code in CPAN modules over the last couple of weeks, followed by a conversation in the CB. Not that I dislike XS per se, the idea of being able to glue in libraries from other programming languages is nice. My personal opinion is that it is overused in CPAN modules, but I'm prepared to be gainsaid on that one. However, I'm constantly finding myself attempting to install modules, following up all the dependencies, and somewhere in the dependency chain XS rises up and bites me.

Yes, perl is written in C. But perl is also shipped, precompiled, with a wide variety of operating systems, which don't necessarily have C compilers installed by default.
Yes, gcc is available for just about every platform imaginable, but not all companies will permit its installation.
Yes, if you're running under windows and the module you want is available from ActiveState, you don't have the compilation problem.
Yes, I know that the language a module is written in is noted on CPAN (assuming that data is up to date) but that doesn't cover the modules dependencies

BUT
If you acquire open source C libraries (openssl for example), you must have access to a C compiler, or they'd be no use to you. Likewise pretty much all other compiled languages. But that isn't true of perl.

My feeling is that any module requiring a C compiler is leaning more towards being an extension to the perl language than being a reusable code library. I'm not suggesting that XS/C modules should be banned, frowned upon or anything like that, just commenting on a perl issue that causes me problems.

The obvious answer is the one suggested to whining travellers in Heinlein's 'The Moon is a harsh Mistress' - "if you don't like it, build your own", and where time allows and there is a need, I'll happily write pure perl versions of existing functionality. But as a passing point if I or anyone else is going to do that, it would be useful to have a standard way of denoting it. For example:

There isn't really a simple solution to the overall problem. For example, recently I needed to encrypt/decrypt passwords. There are pure perl crypt implementations Crypt::Blowfish_PP for example, but Crypt::CBC needs Digest::MD5, which needs C. Of course, Digest::MD5 is often installed as part of a precompiled version now, but on this old version of perl it wasn't installed (I didn't discover Crypt::ECB until later).

So the module dependencies for different modules vary between different versions of perl - there's potentially no such thing as a module which is pure perl in itself and all it's dependencies.

So, thoughts/observations/flames anyone?

Update: Just as a final observation, the title of this node (Nicked from Martin Luther) contains a mention of power. Over at comp.ai.neural-nets, where they're really into their complicated calculations, someone reckoned perl to be 20-30% slower than C for neural nets. I don't think thats so very bad, myself....

Update: I figured this would be contraversial - hence the Luther references. There seems to have been a degree of misunderstanding though. I'm not anti C or anti XS - in fact I use C for other purposes in other environments. XS oftentimes causes me pain because it frequently insists on the same compiler for perl & module though, a limitation that, while I understand it, can present difficulties.

And yes, sometimes a c compiler is not available for whatever reason, and while saying 'thats just stupid' may be true, it doesn't help solve the problem.

Primarily, I was, and am, interested in what other monks have to say on the subject. Thanks to all those who've already replied.

g0n, backpropagated monk
  • Comment on Disputation of g0n on the power and efficacy of XS

Replies are listed 'Best First'.
Re: Disputation of g0n on the power and efficacy of XS
by brian_d_foy (Abbot) on Apr 20, 2005 at 17:21 UTC

    It sounds like your problem is access to a C compiler, not authors using XS. That problem isn't the availability of a C compiler for your platform, however. It's a social problem outside of technology.

    Perl is an open source community and code re-use is rampant and appropriate. I'd rather see a module use a widely used and well maintained C library than reimplement it and fall into disrepair. You mention openssl: when the C library that most people use has a new version, I don't have to wait for a Perl version to catch up. I don't have to watch parallel evolution in code that does (or should do) the same thing.

    Every module is an extension of the Perl language, and they are all re-useable code libraries. The implementation details are the red herring here. Although it would be nice to have a way to select different implementations of things at a low level, and the programmer level I just want to use the same module name everywhere and know I'll get some implementation of it. If the local machine has the C implementation, I get the speedy C implementation. If it has the pure Perl implementation, I end up using that without changing my script.

    This has never been a big deal for me because I just get the C compiler, and point out to clients the difference in paying more money and waiting longer for a pure Perl implementation versus downloading a compiler. You don't have to compile it on the target machine either, and with things like ActiveState's PPM, you don't have to compile it yourself a lot of the time.

    --
    brian d foy <brian@stonehenge.com>
[reply]

      I think part of g0n's point is just the pain of getting the external libraries. For example, I'm attempting to get XML::Parser compiled on an AIX box to get XML::Twig to work. Now I have to track down Expat for AIX because, of course, Expat isn't included with XML::Parser.

      I may be misinterpreting the original post, but that's one point I read from it.

[reply]

        Those things have never seemed especially hard for me. Just how easy do you want things to be?

        I'm not sure why you have to track down expat: it's not tough to find. There are even binaries for AIX (which I found by Googling "expat AIX" and following the top link).

        I'd rather not have expat as part of XML::Parser. I'll download the latest version and compile it myself. The current version of expat doesn't have to get out of step with the one that would get locked in with a version of XML::Parser. I can even update them separately.

        Installing libraries isn't that hard, and installing a C compiler isn't that hard (although it can be lengthy). Distributing them with the modules isn't going to make that any simpler.

        The original post complains that we even deign to use something written in C, and correctly says that if you don't like it, you can upload the pure Perl version to CPAN. I'd rather just use what other people have already than reinvent the wheel. :)

        --
        brian d foy <brian@stonehenge.com>
[reply]
      It sounds like your problem is access to a C compiler, not authors using XS. That problem isn't the availability of a C compiler for your platform, however. It's a social problem outside of technology.

      Yes, that's exactly my problem. However, there are occasions when companies are not prepared to pay for a license for a c compiler, and not willing for whatever reason (anti open-source, excessive rootkit paranoia etc) to have gcc installed on their machines. Saying that's soooo stupid may be true, but it doesn't get us very far.

      Moreover, in a company the size of the one I'm currently contracted to (>100k employees) policies do not change easily, and certainly not at the instigation of contract developers (who in the company hierarchy rate somewhere below the office cat).

      Perl is an open source community and code re-use is rampant and appropriate. I'd rather see a module use a widely used and well maintained C library than reimplement it and fall into disrepair. You mention openssl: when the C library that most people use has a new version, I don't have to wait for a Perl version to catch up. I don't have to watch parallel evolution in code that does (or should do) the same thing.

      OK, maybe openssl was a bad example, and I quite agree that its better to use a 'widely used and well maintained C module'. I'm suggesting that a) it would be useful for those occasions where compiled C is difficult or impossible to implement to have a pure perl alternative, where possible; but mainly b) that it would make life a lot easier to know when a module has a C/XS dependency. I know that's tricky, for the reasons I set out in my OP - that's why I raised the point for discussion.

      Although it would be nice to have a way to select different implementations of things at a low level, and the programmer level I just want to use the same module name everywhere and know I'll get some implementation of it. If the local machine has the C implementation, I get the speedy C implementation. If it has the pure Perl implementation, I end up using that without changing my script.

      Again, I agree, and I'm trying to do something of the sort in one of my modules, but at the moment I'm engrossed in improving functionality in the pure perl implementation.

      g0n, backpropagated monk
[reply]
        it would make life a lot easier to know when a module has a C/XS dependency.
        That's pretty trivial, isn't? Configure CPAN.pm to automatically follow dependencies. Do a perl -MCPAN -e'test Module'. If it has a C/XS dependency, it fails. If it doesn't, it either success, or some other error pops up.

        Yeah, it takes a few minutes more than looking it up a table. But it's hardly more work, and you're certain the answer will be accurate.

[reply]
[d/l]
Re: Disputation of g0n on the power and efficacy of XS
by chromatic (Archbishop) on Apr 20, 2005 at 17:59 UTC

    Perl 6 uses Parrot, which has NCI, so it doesn't really have this problem. (NCI doesn't require that you have a C compiler installed to link against C shared libraries, anyway. C++ might be more complex; I don't know enough about its ABI to say.) It's even easier if your platform as JIT support.

    It's possible to solve this problem in Perl 5 as well, though I haven't had time to work on it as much. See P5NCI.

[reply]
      That's one part of the solution--the part that deals with legacy libraries. The other part of the long-term solution is to make sure Perl 6 is a language in which you can express low-level concepts so you don't have to drop into C just for performance. ParrotOS, here we come... :-)
[reply]
[d/l]
      C++ might be more complex; I don't know enough about its ABI to say

      there is no such thing as an standard C++ ABI, it's compiler specific!

[reply]
        Not only is it compiler specific the C++ ABI is usually very fragile. The good news is that the GCC team has done some effort to alleviate the fragility of the ABI on C++
[reply]
Re: Disputation of g0n on the power and efficacy of XS
by samtregar (Abbot) on Apr 20, 2005 at 18:55 UTC
    I think the solution to this problem is clear, you should write a C compiler in pure Perl.

    Oh man, I'm funny huh? But wait, I have a real answer too!

    You mentioned that one reason to avoid XS is company policies forbidding compilers on some machines. I've been there myself and the result was the Krang build system. Krang relies on XS modules the DBI and XML::SAX::ExpatXS but needs to be deployed on compiler-less targets. We created a system to build binary releases of Krang which require only Perl and MySQL to function.

    Maybe you can do something similar. Surely your company doesn't ban compilers on ALL machines!

    -sam

[reply]
Re: Disputation of g0n on the power and efficacy of XS
by cbrandtbuffalo (Deacon) on Apr 20, 2005 at 16:22 UTC
    My thoughts center on a related, if less anti-C, concern. I don't mind that modules require external dependencies, even complex ones. But I completely agree that it would be nice if there was a consistent way to flag them as such.

    I am currently working on a perl upgrade project in our environment and part of this upgrade is tracking all the modules we have. The CPAN module does a great job with this with its 'autobundle' and 'check for new versions' features. But what it can't do (that I know of) is track the same info for these external dependencies.

    So I'll either have to manually go through and catalog this info, or brute force install each new module and let it bark when the dependencies aren't up to snuff.

    If this info was somehow easily available in a standard way from distributions, you could potentially run searches in CPAN, down the dependency tree, to see exactly what would be required.

    Another issue is where this external code has to go for various distributions. Some external dependencies need to be installed outside the module install, and it can be difficult to track the locations of these external components.

[reply]
Re: Disputation of g0n on the power and efficacy of XS
by grinder (Bishop) on Apr 20, 2005 at 16:47 UTC
    it would be nice to know easily beforehand whether a module has a direct, or indirect dependency upon XS or C

    Backe in ye olde darke ages, such information was provided by the DSLIP code of a module, as it appears on The Module List.

    These days, that list has fallen into a state of disuse, and dslip has fallen by the wayside. Many authors are unaware of its existence, never mind your average Perl user who just wants to get the job done.

    The current feeling is to expand the META.yml file that should exist in a reasonbly well maintained module, to provide a revamped XXI century version of the information that used to appear in the old DSLIP codes. There was a thread recently on the matter on p5p (I think, or maybe module-authors) although no firm conclusion was reached. The options for Support were considered especially outdated.

    - another intruder with the mooring in the heart of the Perl

[reply]
Re: Disputation of g0n on the power and efficacy of XS
by perrin (Chancellor) on Apr 20, 2005 at 18:29 UTC
    Popular operating systems usually have binary packages available for all commonly used modules.

    In general, I'd say that providing a nice API to good C libraries is one of the primary reasons for Perl and one of the things it does best. Database libraries, encryption libraries, image manipulation libraries, etc. are available to me in Perl because of this ability.

[reply]
Re: Disputation of g0n on the power and efficacy of XS
by Anonymous Monk on Apr 21, 2005 at 08:55 UTC
    But perl is also shipped, precompiled, with a wide variety of operating systems, which don't necessarily have C compilers installed by default.
    But Perl itself isn't distributed precompiled - that is, whenever p5p makes a new version of Perl available, it makes the source available. Not binaries. To quote http://www.perl.com/download.csp
    Generally speaking, CPAN doesn't distribute Perl binaries. It does provide information in the /ports directory that will point you to sites that maintain binary distributions for specific platforms.

    I think you are in a situation that isn't to be solved by Perl. If it's your own box, install a compiler, or get a different OS. If it's work, explain to your manager why getting a compiler benefits the company. Saving your time means saving money. Saving money means a bigger bonus for your manager.

[reply]
Re: Disputation of g0n on the power and efficacy of XS
by kiat (Vicar) on Apr 21, 2005 at 03:49 UTC
    Not really sure if this is related to the parent thread...

    Recently, I installed Freebsd and spent some time exploring it. I like the idea of using ports for installing packages through the Net but...

    When you try to install X which depends on A which depends on B which depends on a newer version of C than the one currently on your system which depends on D but it's not available for download for some strange reasons...Error at blah blah blah.

    I think the idea of perl modules being totally dependent on perl sounds cool to me.

[reply]

      The BSD you're using should also keep copies of the required source in the distfiles directory on their FTP site. Also, you can usually download this stuff precompiled from your BSD's web site and don't even have to deal with compiling source code.

      So the above isn't a problem that takes more than a minute to work around.

[reply]
Re: Disputation of g0n on the power and efficacy of XS
by Anonymous Monk on Apr 21, 2005 at 09:01 UTC
    Yet it means that perl should be flexible, not necessarily that perl should become a glorified API to C libraries
    I'd say that for Perl to be flexible, it is necessary (but not sufficient) to be able to access C libraries. It's a glue language. Many libraries are written in C - to connect to a database, or to manipulate images for instance - and are written with a much broader scope than "Perl programs" in mind. Perl should, and does my means of XS, provide access to those libraries.

    Of course, you're always free to provide Perl only implemenations of XS modules.

[reply]
Re: Disputation of g0n on the power and efficacy of XS
by fireartist (Chaplain) on Apr 21, 2005 at 08:22 UTC
    My feeling is that any module requiring a C compiler is leaning more towards being an extension to the perl language than being a reusable code library.

    Who told you that cpan modules were simply reusable code libraries?

[reply]
Re: Disputation of g0n on the power and efficacy of XS
by zentara (Cardinal) on Apr 21, 2005 at 11:48 UTC
    XS is not the problem here, MSWindows is. XS modules compile painlessly on linux. Maybe you should spend some time switching to a better platform?

    I love the XS modules for the extra speed they provide.

    Windows is all about making money. If you want to fork over $500 for their spiffy c compiler, it will work nicely. If you want to use the "free versions" you can get, it becomes a nightmare.

    The best computing descision I ever made, was to dump windows, and learn linux.

    I'm not really a human, but I play one on earth. flash japh
[reply]
      What makes you think I'm running windows? There are plenty of commercial unix variants that do not come with C compilers (including AIX, the one I'm using). In point of fact, my personal choice of OS is Linux, and I run it on all my own machines. My customers, on the other hand, do not.

      g0n, backpropagated monk
[reply]
        Ok, I guess I jumped to a wrong conclusion, since most whiners about XS seem to come from Windows users. So then the question becomes "If it's such a PITA, why do you support AIX ? "

        I don't think you have made any points against XS, and I doubt if module writers are going to go through the hassle of making "special pure perl ports" of their XS-based modules, just so a few people can avoid the hassle of installing a C compiler on their platform(Well maybe if you pay them). If you are doing this for some customers, who won't change, explain the situation to them, and charge them more money for "YOU" to write the pure perl ports of the modules.

        Also what puzzles me, if you advertise your services, to support AIX, why don't YOU buy a compiler for AIX, and compile the XS modules for your AIX customers?

        I'm not really a human, but I play one on earth. flash japh
[reply]
      If you want to fork over $500 for their spiffy c compiler, it will work nicely. If you want to use the "free versions" you can get, it becomes a nightmare.
      Untrue, and I should know, I got'em both, no difference, same ease :)

      MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
      I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
      ** The third rule of perl club is a statement of fact: pod is sexy.

[reply]

Back to Meditations