“... If I have seen a little further, it is by standing on the shoulders of Giants.”
– Isaac Newton

This, to me, is one of the most important consequences of the Internet age.   The availability of a world-wide, largely unconstrained data network (and powerful search engines) made it possible for software developers around the planet to build up powerful software foundations, the likes of which the industry had never seen before, and to do so cooperatively.   In the Perl community, the CPAN library has, at this moment:   73842 Uploads, 23837 Distributions 101779 Modules, 9364 Uploaders.

What does this mean to us, as software developers?   Well, it means that we are no longer actually “developing,” i.e. “from scratch,” nearly so much stuff as we once did.   Tangentally, it means that we should be spending a lot more of our time searching for “prior art,” before we attempt to create something altogether new and (we hope and expect) instead of doing so.

Okay, okay ... “duh, we all know that.   What’s your point?”

My point, reaching deeper, is that I think that this demands a greater sense of software-development discipline from us, with regards to the stuff that we aren’t grabbing from some library someplace.   We need to take some lessons from these modern packages, and from CPAN itself.   We need to study not only what the most recent and popular packages do, but how they are built and how they are deployed.   We need to take a peek at the source-code links sometimes.   And, we need to look at those mountains of self-tests.   We need to “go and do likewise.”

Any CPAN package, to pass muster, must be provided with certain things:   embedded documentation, a thorough prerequisites/corequisites list, a rugged overall construction, and very thorough and complete self-tests.   This, and only this, is what created the vast library of modules that we now enjoy and that we drop into our critical projects with relative impunity.   We need to push ourselves, and our teams, to that standard of excellence.

Do all such modules achieve that standard?   No.   But they (should) aspire to it; they are expected to.

When we do this, we achieve a level of human performance (in terms of “quantity of error-free functionality generated” as well as in terms of “maintainability of that base”) that has sometimes been referred to either as haiku or Zen.   Perhaps a better way of describing it is “software leverage.”   Even a single tough, well-engineered and well-constructed software module can have dramatic positive impact upon a project.   Surfacing in many places throughout a project, and utterly (provably...) dependable in whatever it does, the module brings reliability and consistency at the same time.

Replies are listed 'Best First'.
Re: "Standing on the shoulders of giants"
by JavaFan (Canon) on Nov 28, 2011 at 21:26 UTC
    The one thing that made CPAN a success was no entrance barrier. All code is welcome on CPAN. Documentation is not required. Prerequisites are not required. Rugged overall construction is not required. Tests are not required. Aspiration is not required. Approval by some Perlmonk is not required either.

    The result is that there's a lot of crap on CPAN. And that the existence of a CPAN module doesn't mean anything, other than that the author has figured out how to use PAUSE.

    But this makes CPAN work. No requirements. No pressure.

[reply]

      What would make CPAN work even better would be to encourage not so effectively discourage collaboration. CPAN right now does an excellent job of discouraging collaboration. The funniest part is the case when real collaboration isn't even required; the most common case of free software development; the case of the person eventually moving on after they scratched their itch; the case where CPAN does an amazing job of discouraging simple maintenance.

      It makes associating the phrase "on the shoulders of giants" with CPAN humorous to me. That's the aspect of good software development that CPAN does the best job of preventing.

      - tye        

[reply]

        Agreed; CPAN is a lot more useful for me when tools like Git make forking and patching so much easier. Eric Wilhelm's SVN for CPAN would have been nice several years ago.

        Obviously the responsiveness of maintainers is still important (and perhaps my worst failing as a CPAN author) but actively encouraging collaboration would be amazing. To the credit of many people, CPAN has improved dramatically in this respect as of late.

        Improve your skills with Modern Perl: the free book.

[reply]
        I had never thought from this angle. Real collaboration is not required and still it is going on smoothly.
[reply]

      But of course there is pressure. Quite a bit actually. Peer pressure.

      While there's nothing technical or official preventing you from uploading utter untested and undocumented crap, you generally try not to do that. Because of the way you'd look among the others. So if others add tests, you do too. If others document, you do too.

      The sole fact that you make something public is a pressure enough.

      Jenda
      Enoch was right!
      Enjoy the last years of Rome.

[reply]
        Really? People write tests and documentation just because others do, not because they think it's a good idea all by themselves? It's just like a religion then, you do stuff because others do so, not because you have two braincells to rub against each other.

        You have an even lower opinion of the average CPAN author than I do! Congratulations, I didn't think that was possible.

[reply]
Re: "Standing on the shoulders of giants"
by locked_user sundialsvc4 (Abbot) on Nov 29, 2011 at 03:37 UTC

    And then, there is the delicate matter of religion ...

    The one thing that Monks, being Monks, most love to talk about.

    ;-)

    But anyhow ... to all of those of you whose work, so much of my work for so many years so happily requires ... you are “giants.”   And I thank you all.   Very much.

    Just sayin’ ...     @>--+---

    (Now, would you please consider adding just one small, eensy-weensy feature ?? ...)

Re: "Standing on the shoulders of giants"
by Anonymous Monk on Nov 29, 2011 at 00:57 UTC

    There is just one problem with that; yesterday's solution is not necessarily the best for tomorrows hardware. (or user requirements)

    Various folks seem to think that software once written never needs to be rethought, and whilst that may be true one day; when we have completely and finally optimised and maximised the hardware side of the equation, currently it is not true and it's a mistake to think that we can rest on our collective laurels, treat existing code like gospel beyond question, and to dismiss out of hand anything new.

[reply]
      Various folks seem to think that software once written never needs to be rethought

      What a ridiculous supposition.

      In my experience, there are no shortage of people who announce their new software and then get questioned about why they went to so much effort rather than just using this or that similar pre-existing software. And they mostly get asked that because their announcement of their new software fails to answer such questions (usually it fails to even acknowledge the existing similar solution much less make a good case for why the new solution is better for the motivating case, much less "in general").

      This is so common that it has become quite common to start with the assumption that insufficient consideration was given to the option of not starting from scratch. I admit that I would prefer some people to moderate their devotion to evangelizing around that.

      It seems to me that a more probable supposition is that some people get tired of having their from-scratch creations questioned and so invent a ridiculous pathology to help them dismiss such responses.

      I even reinvent a lot of stuff. And I see a lot of skepticism that some reinvention of mine was worthwhile. If it bothered me, I'd put more effort into documenting the benefits of my versions as compared to others. But I probably also see less skepticism than many because I often feel little compulsion to evangelize my (re)creations. Go boldly bragging about your awesome (re)invention and you'll get extra helpings of counter points, naturally.

      - tye        

[reply]
        My #1 reason for "reinventing the wheel": it's often faster to just do whatever scratches my itch than to find out whether someone already has done it, decide whether it's actually good enough for me, and to figure out what hellish API they may have thought up.
[reply]

        I freely and openly admit that I wrote my 're-invention' because when I started it I had absolutely no knowledge of what had been before, not even CGI.pm

        But once I'd started down the path I found, the more I learnt about what I didn't know, the more I found that I didn't like it anyway.

        I also freely admit that a lot of the problem I encountered in presenting my ideas was my own fault for not properly documenting what I had done and for getting annoyed at people failing to understand it because I hadn't documented it.

        To be honest I'm one of these people that flits willy nilly from one thing to another, I have attention deficit and if I'm not coding, I'm playing a game or my guitar, or piano, or I'm outside having a smoke.... anything but writing documentation!!

        At the end of the day I'm 100%, no 110% glad that I took the path I took and came to where I am now :

        http://www.perlnights.com/?action=show_thread§ionid=1&threadid=251

        I guess I really ought to write the documentation, and find a better name for the code which is not in use by someone else. I'll do that eventually, right now I can't be bothered :)

[reply]

Back to Meditations