http://qs1969.pair.com?node_id=573100

Update: I suggest you first looking at solution below, before reading complete thread.

Class, prepare your mouse pointers, take them near '--' and be ready to click 'Vote!' because I'm going to say something terrible about CPAN.

Few days ago I've asked about Reliable email parsing and that discussion uncover some points:

For me, ANY software MUST be developed with these priorities in mind:
  1. be reliable: it should do it work correctly (ex. for email parsing task it must support all features defined in RFCs related to email format plus be able to handle non-RFC-compliant emails produced by some buggy software)
  2. be secure: it should not allow unauthorized usage
  3. next priorities may vary from have a lot features to work fast or be cheaper or have intuitive interface, etc.
... because if software isn't doing exactly what it should or open way for hackers/viruses into my system I don't need it, no matter which (unreliable!) features it have!

I say "ANY software", but especially this important for all reusable things like perl modules and core things like OS kernel, perl itself, text editor or web server (i.e. all things which you use to develop your own application). Without this developing own reliable application become a nightmare because lower layers which you use isn't reliable itself and you spend huge amount of time detecting and fixing bugs there... or reimplementing these layers as part of your application. :(

Now about CPAN. 99.9% of CPAN modules developed without reliability and security in mind. They are "worksforme", feature rich, have ugly interfaces, bloated, clever, anything! But not reliable and secure.

So, if you wanna develop reliable application you can't use most of CPAN modules. List of "safe" modules may vary, but usually it include:

If this looks like "a lot" of modules then check them again: they all are low-level modules, each doing small simple task (probably except DBI)! All CPAN modules I seen which try to solve more complex tasks doesn't looks reliable and secure. A-L-L! Ok, I (still) hope there some exceptions which I missed, but this doesn't important.

So... for every high-level task (and low-level tasks which has no good enough CPAN module) you must develop custom solution. We all read: 'CPAN is goodness!'; 'Any your task already solved by reusable CPAN module, no need to reinvent it!'; 'Perl is better than other languages because only Perl has CPAN!'; etc. for YEARS...........................
This result in people stop thinking critically about CPAN, they believe is't goodness because it IS goodness, and that's all.

Now it's time for adducing some proof, small examples for people who think CPAN modules ARE reliable.

First example - executing external process task.
There already exists a lot of ways to execute command: system(), open(), `` (backticks), IPC::Open3, IPC::Run, IPC::Run3. In short, system() is good but doesn't allow interaction with running command, and all others doesn't handle signals correctly. Below is gore details, if interested.

While it's possible with system() to run command with custom filehandles instead of inheriting only STDIN/STDOUT/STDERR of current process, it has some limitations: - You can't use safe LIST form of system() and should execute your command using shell to have filehandle redirections like '2>&1' working. - You can't interact with running command using pipes. - If you need to give your non-STDIN/OUT/ERR filehandle or if you need to give more than 3 filehandles to command you should use fcntl() to modify close-on-exec flag and/or reopen your STDIN/OUT/ERR to needed filehandles. - User can't set timeout for command. There no way to setup alarm() for command. User can setup alarm() for his main process, but: 1) this isn't acceptable in module because user can already setup some alarm() before calling our module 2) this alarm() will interrupt system(), but not interrupt running command (and we can't kill it because we don't know it's pid) open() limitations: - open(), unlike system(), doesn't block SIGCHLD. This may be fine in user code, but in module this result in two problems if user has $SIG{CHLD} handler installed: 1) pid of process open()'ed in module will be delivered into user's SIGCHLD handler 2) $? status will not be available in module - open(), unlike system(), doesn't block SIGINT & SIGQUIT in main process so if you use open() instead of system() just to interact with executed command using pipe like: open(my $fh, 'some_foreground_command |') then both this command and your main process will receive these signals if user press Ctrl-C or Ctrl-\. - If command will exit while user print() into pipe SIGPIPE will kill main process if user don't block it. - All system() limitations also apply to open(), with only correction: you can interact with running command using pipe, but only single pipe - if you need more pipes then you should use IPC::Open3, IPC::Run or IPC::Run3. `command` (backticks) limitations: - No safe LIST form. - You can't interact with running command using pipes. - If you need to give your non-STDIN/OUT/ERR filehandle or if you need to give more than 3 filehandles to command you should use fcntl() to modify close-on-exec flag and/or reopen your STDIN/OUT/ERR to needed filehandles. - All open() limitations apply. IPC::Open3 limitations: - No signal handling at all, so all open() limitations about SIGCHLD, SIGINT, SIGQUIT and SIGPIPE apply. - No timeout. - Unable to use more than 3 filehandles. IPC::Run limitations: - No signal handling at all, so all open() limitations about SIGCHLD, SIGINT, SIGQUIT and SIGPIPE apply. IPC::Run3 limitations: - No signal handling at all, so all open() limitations about SIGCHLD, SIGINT, SIGQUIT and SIGPIPE apply. - You can't interact with running command using pipes.
My realization of this task have 110 lines of code and it's only realization I know which handle all nuances described in Stevens APUE book. You can check it by downloading POWER::Utils module from my website.

Second example - timers and timeouts.
If you use time() function for realizing timer or timeout, no matter CORE::time() or Time::HiRes::time() -- your code is unreliable. Why? Because. Because there NTP and /bin/date. And they may (and will!) change current time, both forward and backward. Only reliable way to realize timer/timeout in Linux - use CLOCK_MONOTONIC param for clock_gettime(2). This required syscall() until I've asked Time::HiRes author to add this feature, and since Time::HiRes 1.77 you've clock_gettime(CLOCK_MONOTONIC) feature. How many CPAN modules work with timers/timeouts? And is even single of them use CLOCK_MONOTONIC? Few months ago I've searched CPAN and found 0 such modules.

Third example - mailbox parsing (from my previous thread).
In short, mailbox can be in one of 4 formats: mboxo, mboxrd, mboxcl and mboxcl2. There no way to autodetect it. Reading from mailbox using wrong format lead to damaging messages. CPAN has many modules which has 'read mailbox' feature, but no one of them allow user to configure mailbox format BEFORE reading, and only one has note in documentation about these formats (and try to do it best autodetecting it, which is anyway impossible).

Fourth example - reliable eval().
Eval? What's the hell is wrong with eval!? Only one: eval() doesn't support one advanced perl feature: source filters. You can found my version of eval (6 lines) which compatible with internal perl eval but also support source filters in same POWER::Utils module.

Want more? Ok. Fifth example - using GPG.
CPAN contains a lot of modules for accessing GnuPG, but they all not reliable enough and some of them not secure. My version (see POWER::GPG module on my site) is module execute gpg with correct signal handling; use non-blocking pipes and multiplexing to avoid hang on processing huge files; never store sensitive information on hard drive in temporary files (one temporary file used for checking detached signature); use timeout to protect against gpg hangs. Details about existing CPAN modules below:

Limitations of other modules from reliability/security view: =over =item GPG - Doesn't handle SIGCHLD, SIGINT, SIGQUIT. - Hang on large files. - Parse unreliable STDOUT instead of reliable --status-fd. =item Crypt::GPG - Doesn't handle SIGINT, SIGQUIT. - Incorrectly handle SIGCHLD. =item GnuPG - Doesn't handle SIGCHLD, SIGINT, SIGQUIT. - Use deprecated shared memory interface. - Use temporary files to store sensitive information. =item GnuPG::Interface - Doesn't handle SIGCHLD, SIGINT, SIGQUIT. =back

And the last thing which drives me crazy. Perl isn't a new language, it activelly used in the world for more than 10 years. Perl, by design, is text processing language, used initially for system administration. Parsing emails is really needed task, it's text processing task and it's related to system administration. CPAN has huge amount of modules for this task, and no one of them support all email-related RFC. After so many years no one developed such module! AND NO ONE THINK IT'S NEEDED - at least comments to my question say so......... "Yeah, why you think you need reliable email parser, what you will use it for? Forget about it, man, you don't need it. Nobody need it!" :(

Replies are listed 'Best First'.
Re: Reliable software OR Is CPAN the sacred cow
by davorg (Chancellor) on Sep 15, 2006 at 12:33 UTC
    This result in people stop thinking critically about CPAN, they believe is't goodness because it IS goodness, and that's all.

    I don't believe that this is an accurate summary of anyone's views on CPAN (or, at least, anyone who has looked seriously at CPAN).

    There is a _lot_ of code on CPAN. A lot of it is of very high quality and very useful. Some of it isn't. I don't think that's a problem with CPAN however, I see the low barrier to entry for CPAN as a feature. The Perl community has, in recent years, set up side projects to CPAN which allow users to see the "quality" of CPAN modules (as discussed by other users or determined by automated tests) and to give them more information on which to base their decision on whether or not to use a particular CPAN module.

    We all realise that there are problems with some CPAN modules. There are almost certainly still bugs to find in pretty much every CPAN module. When you find one of these problems, you have a choice. You can mutter vaguely about the problem (as you've been doing on this site for the last few days) or you can help the author of the module to fix the problem and thereby help make the module better. There are a number of ways that you can help:

    • Raise a bug report in RT (it would be nice if your bug included a failing test).
    • Review the module on CPAN Ratings
    • Annotate the POD to correct any error
    • Join the module's mailing list (if one exists) and explain the problem
    • Offer to fix the problem - provide a patch

    This is open source software. Of course, you're free to use it (or not use it) as you wish. No-one is going to force you to contribute. But it would be really good if you could help out. You've made it clear recently that you're not happy with the quality of CPAN's email handling modules. But as far as I can see, you have done nothing to improve the situation. Yes, you say you've written your own implementation, but why haven't you released it? Or why haven't you contacted the Perl email project to offer your help? Or given them concrete examples of the way that their current modules fail? It's very easy to complain, but harder to do something constructive.

    I don't believe that CPAN is a sacred cow. And I don't know anyone who does. I know that there's an awful lot of rubbish on CPAN. But I find rubbish there that I'd like to use then I don't waste time moaning about it, I do what I can to fix it. Why don't you do the same?

    --
    <http://dave.org.uk>

    "The first rule of Perl club is you do not talk about Perl club."
    -- Chip Salzenberg

    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Reliable software OR Is CPAN the sacred cow
by perrin (Chancellor) on Sep 15, 2006 at 12:39 UTC
    Well, duh. It's very rare for two people's requirements to be exactly the same, so of course the existing modules don't all do exactly what you wish they did. Your definition of "reliable" seems to be "does precisely what I want."

    This is the part where you're getting people mad at you. If you run around shouting that people's code is "unreliable" rather than simply not covering some functionality you want (Source filters in eval? Give me a break!), you are certain to annoy them and make them less likely to help you. When you add injury to insult by not offering to help improve what you say is lacking in their modules, what do you expect them to do? Thank you and beg for more of your great wisdom?

    CPAN comes with a 100% money back guarantee. Feel free to cash in, or to fork some modules, or to help people fix modules, or whatever it is that you want to do. Meanwhile, the horribly broken e-mail modules that you're complaining about will go back to running most of the Internet's mail.

    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Reliable software OR Is CPAN the sacred cow
by talexb (Chancellor) on Sep 15, 2006 at 12:20 UTC
      For me, ANY software MUST be developed with these priorities in mind:
      1. be reliable: it should do it work correctly (ex. for email parsing task it must support all features defined in RFCs related to email format plus be able to handle non-RFC-compliant emails produced by some buggy software)
      2. be secure: it should not allow unauthorized usage
      3. next priorities may vary from have a lot features to work fast or be cheaper or have intuitive interface, etc.

    So, what's your thesis? Are you complaining that there are no Mail::* modules that follow 100% of the RFCs? Or that 99.9% of the modules on CPAN aren't reliable, secure, etc.? Or both?

    Well, this is just a reminder that this is the open source community: We're mostly volunteers. If you wanna write something that'll help you get your job done better, more efficiently, more safely, and if you want to sure that work with everyone else, terrific. If you want to use something that someone else has written, go for it. You can even make your own improvements and maybe even get those improvements included, with the thanks of the authour and the community.

    But if you're complaining that none of (for example) the Mail::* modules are totally compliant with the RFCs and want to blame someone, blame yourself for expecting the perfect solution for free. And if you want to do something about, use one of the existing modules and fix it up (see previous paragraph) or write your own, using the RFC. You may not encounter fame and fortune as a result, but if the module works well, the community will thank you and you may find yourself being bought a lot of beer.

    Finally, the requirement that a CPAN module have an intuitive interface escapes me a little -- you learn about the module interface by reading the module documentation. I don't know how a module's interface can be made .. intuitive. For example, DBI uses the 'connect' method to create a new database connection object. By the time someone pointed how non-OO this was (of course, it should be a 'new' method, everyone knows that), I'd already been using the 'incorrectly name' call to DBI for years, and thought nothing of it .. but it sure is intuitive.

    I'm working on preparing a module for CPAN right now, and it's hard .. because I haven't done it before .. but I thnk it'll be worth it, both for myself and for anyone out there who might use it.

    And boy, I'm looking forward to that beer. :)

    Alex / talexb / Toronto

    "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

      Well, this is just a reminder that this is the open source community: We're mostly volunteers. If you wanna write something that'll help you get your job done better, more efficiently, more safely, and if you want to sure that work with everyone else, terrific. If you want to use something that someone else has written, go for it. You can even make your own improvements and maybe even get those improvements included, with the thanks of the authour and the community.

      Note - I'm not specifically addressing talexb in this reply, but the OSS community in general.

      I think this attitude is a cop-out, since it is exactly counter to one of the original points about CPAN being a sacred cow. There are innumerable replies on this board and others with people saying, "Don't re-invent the wheel! Use Foo::Widget from CPAN!" And that's fine, we're allowed.

      But jumping to "Hey, it's free software and you get what you pay for!" if people gripe about it doesn't help the case for open source at all. Yeah, I know, we are all volunteers (well, mostly), and yeah, people do get what they pay for, but it's a horrible way to try and defend yourself.

      How many people around here gripe about Microsoft's ever increasing systems requirements for its operating system and office suite? I need a 8GHz Teraflop 90 core Whatsamadoodle just to run a word processor? That's BS!. And that's fine, gripe. But then how many people turn around and say that perl is a lightning fast language and it's just as good as C for most tasks (at least on the web) and things would just be a-okay if we threw some more hardware at it? If Microsoft gets chastized for ever increasing requirements, so should you.

      Same with open source. You can't sing the virtues of the wide open community with its own source code and legion of people poking the code and then turn around and say "We're all volunteers. That's what you get!" It firmly and completely negates the original benefits, I think.

      A previous boss used to really sneer about using "free software" for these reasons. The app we had was written in an old version of Mason that wasn't compatible with Perl 5.8. But, the API had changed enough that it would require some (unknown but probably semi-big) amount of work to upgrade to a current Mason. So the app is stuck on perl 5.6 and old mason. When he came on board and heard about this, the first thing he blamed was free software and how you can't trust anything and no one cares.

      Side note - I did point out to him things like .NET's API radically changing or Oracle support nightmares or whatever. Commercial software doesn't necessarily give you any more stability, reliability, or continuity. You just pay more for it. He wasn't convinced.

      I feel like I'm starting to ramble, so I'll close up. But I don't think that falling back on "It's free, it's what you get" helps the case.

        The point, as I see it, is that open source is a possible starting point in the journey of software development.

        No one's obligated to use open source. You are welcome to write your own stuff from scratch if you want. For me, however, open source is a heck of a good place to start, and I accept the reality that something I get from CPAN may not be complete, or bug-free, with the knowledge that I can make a difference by helping improve the module I'm using.

        And that's an important difference. I can make a change to Template::Toolkit if I want, for example. If I find a bug, contact the authour/maintainers, talk with the folks in the community, offer the patch, how can that not benefit everyone? I get something that works 1% better. So does the community.

        As everyone donates their 1%, we get a better and better product, and build a stronger and stronger community. Many eyeballs make for shallow bugs, and all that.

        Instead of fellow monk powerman ranting about how poorly the Mail::* modules fare in following the various RFCs and in security and reliability, I believe s/he should concentrate their energies on making it right.

        Alex / talexb / Toronto

        "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

        But jumping to "Hey, it's free software and you get what you pay for!" if people gripe about it doesn't help the case for open source at all.

        It doesn't help the naive case, but I don't care one whit about the naive case for open source or free software.

        The reason I believe that open source and free software can reach higher levels of quality and utility and applicability than proprietary software is because open licensing gives a wider community the opportunity to improve the software.

        The real response really ought to be "If you're part of the community, the quality and applicability and utility of the code is your concern too. Contribute." Contrarily, if you don't contribute in some means, you're very clearly not part of the community and your opinion about the code doesn't really matter.

        But jumping to "Hey, it's free software and you get what you pay for!" if people gripe about it doesn't help the case for open source at all. Yeah, I know, we are all volunteers (well, mostly), and yeah, people do get what they pay for, but it's a horrible way to try and defend yourself.
        I'd like to disagree, if only partly and on the interpretation. The Perl community has one big advantage: CPAN. But with this I don't mean the large (huge) base of code, but the people providing the community with it. CPAN lives from contribution, from critique, from honest opinions and bug reports. But it also feeds on the willingness of people to contribute. In this context, I find a comment that reads mostly like "Don't upload your crap to CPAN! I can't use that in my business!" pretty disrespectful.

        Nobody forces anyone to use any module from the CPAN (Well, your contract with your employer might, but you've signed it and took the job). You can use only those tools for the job that you find the right for the job, and it's up to you to either share the authors and communities opinions or disagree.

        In my opinion, the solutions to problem domains provided by CPAN for the workforme category are as invaluable as the larger modules that evolve into a better maturity and bring Perl forward. And in this consequence I'd have to answer the question if people should upload their worksforthem code or not with a loud and clear "Yes." Without these modules and contributions, you wouldn't have a solution either. But with them, someone else might. And who knows? Modules can be taken over, forked and even made reliable by other people if there's a need for it and someone's willing to give their time. That's a pretty important part of open source too.
        Same with open source. You can't sing the virtues of the wide open community with its own source code and legion of people poking the code and then turn around and say "We're all volunteers. That's what you get!" It firmly and completely negates the original benefits, I think.
        Yes but, ... what do you want us to say? It's just the truth. If someone doesn't want to write module XYZ himself, how can he expect anyone else to do it for him? And if he's wanting to contribute the module, what's the fuss?
        A previous boss used to really sneer about using "free software" for these reasons.
        Then I'd say he didn't understand "free software." It doesn't mean "same as if you buy it, but you don't have to pay for it." Freedom always includes the consequences. There is no one true solution.
        Commercial software doesn't necessarily give you any more stability, reliability, or continuity. You just pay more for it. He wasn't convinced.
        Exactly. I don't think the reason there lies in open source quality, tho, but more in peoples understanding of it.
        But I don't think that falling back on "It's free, it's what you get" helps the case.
        Well, to someone asking for more power in some areas of Perl development, I'm afraid it's the only thing I could think of to answer.

        Ordinary morality is for ordinary people. -- Aleister Crowley
    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Effective communication OR Is POWERman the sacred cow
by tye (Sage) on Sep 15, 2006 at 18:37 UTC

    Calling all of your modules POWER::* gets you plonked into the "kooks who don't 'get' CPAN" category. All modules must be well named. 99.9% of your modules have horrid names. Nobody of powerman cares at all about effective communication. A module that is 100% reliable is no good if nobody can find it and nobody wants to use it because the author comes across as a ranting loon.

    Complaining about the quality of what is on CPAN when none of your modules are on CPAN also costs your credibility points. If you think CPAN can be done so much better, then please prove it by actually producing a module that would be admired by the Perl community and could serve as an example of how to do CPAN modules right. But please don't upload modules named POWER::* and really don't upload a module with a totally useless name like POWER::Utils.

    Actually, I'm not sure that you uploading modules to CPAN is a great idea. I'm glad that you put a lot of stress on reliability and identify some features that can be important if you need very reliable solutions. However, reliability isn't the only desirable feature of a CPAN module and I don't think you'd be good at meeting the other desirable features. Good Enlish documentation is required. If all of your modules require POWER::Utils then I suspect you don't have a good enough grasp on modularizing parts of software. Then naming modules isn't an easy task and it is one that is very often not done well enough on CPAN, and I worry that your contributions wouldn't be an improvement on this front.

    Please do contribute to CPAN. But I think it would be better if you contributed to CPAN by, as clearly and politely as possible, identifying where existing modules fail in ways you consider important and working with the maintainers of those modules to get improvements made. I currently get frustrated that CPAN seems to stress ownership too much and contribution too little, so it will require your best diplomacy and communication skills to get your contributions successfully into existing CPAN modules. The best case would probably be if some module authors would let you write code (perhaps provided to them as patches). Alternately, you could find someone to help you with your weaknesses and produce reliable code while they ensure that it is more modular, better named, and better documented so that it would make a good CPAN module.

    I think you raise some important points. But I don't think you've raised them in a way that is going to do anyone any good. Software can't just be reliable. It must also "work for me", else it won't get used and all of that work making it reliable is wasted.

    I agree that CPAN in general would benefit from more attention to issues of reliability. It would be great if you could focus your energy on this point to actually get this improved reliability up on CPAN where people will find it and be able to use it is not just reliable but also "works for them". Doing that isn't always easy. I certainly can find many examples of where I haven't done this.

    - tye        

    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Reliable software OR Is CPAN the sacred cow
by blue_cowdawg (Monsignor) on Sep 15, 2006 at 13:08 UTC
        1. be reliable: it should do it work correctly (ex. for email parsing task it must support all features defined in RFCs related to email format plus be able to handle non-RFC-compliant emails produced by some buggy software) 2. be secure: it should not allow unauthorized usage 3. next priorities may vary from have a lot features to work fast or be cheaper or have intuitive interface, etc.

    Are you trying to tell me that your own code is 100% bug free and does all the above (or some analog thereof) 100% of the time?

    Gimme a break! I know that I've probably written on the order of 1,000,000 lines of code over the last 35 years (proably more than that, I-dunno, the number ain't the point) and I've run into more WTFs in my own code. Subroutines that "shoulda worked" that didn't for one reason or another, forgotten switch cases, token types in a parse I wasn't expecting, you name it.

    You're complaining about something that was written by volunteers. That's silly. If it's broke and you have a fix for it, gently and politely contact the module's author(s) and let them know you have a fix. They might even incorporate it in a future release of the module.

    What I've described in a nutshell is the way Open Source is supposed to work and what is supposed to be the strength in Open Source. This is the difference, at least in my mind, between free, as in "freedom of speach" and free, as in "free beer."

    If all you do is take from the Open Source world and don't at least offer to give back in some form or another then, in my humble opinion, you've blunted your right to complain. Even if your not offering code to fix a problem, you could at the very least offer up to the author(s) your situation as a test case of where the OSS code in this case is going wrong.

    Hey... if code were 100% reliable 100% of the time us System Admistrator types wouldn't have jobs! :-D

    Just my US $0.02 worth....


    Peter L. Berghold -- Unix Professional
    Peter -at- Berghold -dot- Net; AOL IM redcowdawg Yahoo IM: blue_cowdawg
Re: Reliable software OR Is CPAN the sacred cow
by dave0 (Friar) on Sep 15, 2006 at 13:43 UTC
    Wow, what a rant. Given that I have a limited number of hours in the day, I'll ignore most of it, but this part right here:
    Parsing emails is really needed task, it's text processing task and it's related to system administration. CPAN has huge amount of modules for this task, and no one of them support all email-related RFC. After so many years no one developed such module! AND NO ONE THINK IT'S NEEDED - at least comments to my question say so......... "Yeah, why you think you need reliable email parser, what you will use it for? Forget about it, man, you don't need it. Nobody need it!" :(
    needs a response. I guess you miss the entire point of the Perl Email Project, as pointed out to you earlier? The project aims to identify the current best-of-breed email modules in CPAN, and:
    1. Recommend them as the best option for those tasks
    2. Develop and improve them further
    So, here's at least one group thinking critically about CPAN, and working to improve it, in the particular area you're ranting about. The modules aren't perfect, nor will they ever be -- no software is -- but they can get a lot closer, and that's what we're trying to do.

    If you actually want better email modules, it would be a good idea to (politely) file bugs on those modules for whatever it is you claim is broken. Ideally, providing a patch, or a test case, but at minimum a description of the bug and how to reproduce it.

Re: Reliable software OR Is CPAN the sacred cow
by Fletch (Bishop) on Sep 15, 2006 at 13:39 UTC
    AND NO ONE THINK IT'S NEEDED - at least comments to my question say so......... "Yeah, why you think you need reliable email parser, what you will use it for? Forget about it, man, you don't need it. Nobody need it!" :(

    I think you need to re-read the replies a bit harder. They mostly boil down to "Try module X or Y; if those don't work contact the maintainers with test cases which exercise the problems you think there are".

    • davorg: "And if they don't, then the project members seem pretty open to receiving bug reports and patches."
    • davorg: "If you have specific areas where you know the existing modules have problems, then why not get involved with the Perl Email Project and help them fix those problems."
    • dave0: "Well, given that I use quite a few of those modules on a daily basis, I find it hard to believe that they're all "far away" from compliance. Do you have any specific issues to point out?"

    They're not saying live with the bad software, they're saying if you've got problems show how to reproduce them and let the maintainers fix them. If they don't, you're free to put your own überperfect code up on CPAN to shame them in the glorious open source meritocracy that it is.

Re: Reliable software OR Is CPAN the sacred cow
by tilly (Archbishop) on Sep 15, 2006 at 18:04 UTC
    I believe that someone has never read or understood the classic essay Worse is Better. There is also a good followup discussion. Do so.

    As for me, I am plenty aware of the shortcomings of CPAN. For instance one of the modules that you point to as being reliable, Carp, is reliable in large part because I made it so. (And I shudder at how much buggy code there is out there which is crap because it used the old $Carp::CarpLevel interface, which I had to jump through hoops to support because people were misusing it even though it was undocumented.) However I am also aware that most people who come asking for a solution simply won't come up with something better than what is on CPAN. Therefore for them the best answer is to use CPAN.

    Heck, I know that I generally won't come up with something better than what is on CPAN without doing a lot of work. If it isn't really important to me, then I should use CPAN as well. And I usually do.

    If this bothers you, then you should not be using commodity software. I would suggest not using Windows, OS X or Linux. OpenBSD might be up to your standards.

    A random note. You are unhappy that eval doesn't "support" source filters. Well I have news for you. If you care about reliable software, then in any language with a syntax that is more complex than Lisp, you should not use source filters. If you don't understand why, then you are not going to succeed in writing reliable software.

Re: Reliable software OR Is CPAN the sacred cow
by b10m (Vicar) on Sep 15, 2006 at 13:40 UTC
    For me, ANY software MUST be developed with these priorities in mind:
    1. be reliable: it should do it work correctly (ex. for email parsing task it must support all features defined in RFCs related to email format plus be able to handle non-RFC-compliant emails

    POWER::Email is funny though, coming from mr. RFC-is-the-law:

    "Simple (and non RFC compliant) email sending"
    --
    b10m

    All code is usually tested, but rarely trusted.
Re: Reliable software OR Is CPAN the sacred cow
by Albannach (Monsignor) on Sep 15, 2006 at 14:56 UTC
    Just to add my vote, I don't know anyone who treats the CPAN as a sacred cow, and I've yet to hear anyone say that even a single module in the CPAN is perfect.

    However, in my experience it is generally a far far better use of my time to pull pieces out of the CPAN than write them myself. Sure I may find spelling mistakes, bugs, or incomplete solutions, or even security issues, but I'm way ahead of the DIY or commercial options in that I have the source, and I have a community of other users who are generally more than passingly interested in hearing of problems and helping with solutions.

    For the record, I upvoted you on this thread because, despite your somewhat hostile tone, I think this discussion is a good one to have now and again, and I particularly agree that people shouldn't just put all their personal modules on the CPAN in some misguided quest for glory. I'd encourage you to add your best work though. If it is as good as you say, it would be both an important contribution the the community, and a good example of the care that should be taken in publishing modules. And I dare say that some users may offer corrections to your work from which you will benefit.

    I think that this is a case where yelling from outside is less effective than coming inside and lending a hand.

    --
    I'd like to be able to assign to an luser

Re: Reliable software OR Is CPAN the sacred cow
by derby (Abbot) on Sep 15, 2006 at 13:36 UTC

    I never thought of CPAN as a sacred cow ... just the giant who's shoulders I stand on. I think a rite of passage in the Perl Community is when you can separate the wheat from the CPAN chaff. (All right ... that's way too many trite phrases for one post).

    -derby
Re: Reliable software OR Is CPAN the sacred cow
by bigmacbear (Monk) on Sep 16, 2006 at 01:05 UTC

    There is an observation that applies to all sorts of engineering, research, and product development:

    Good. Fast. Cheap. Choose only two.

    What happens is that in the real world there are always trade-offs between quality, development time, and cost. Essentially, there is a finite pool of resources available to a project, and if you need more of one thing, at least one of the other two (and possibly both) will suffer.

    On a commercial project, if you need reliability*, you will have to pay for it either in a longer development cycle or in paying more programmers (or both). If you need working software yesterday, you can hire an army of programmers or skimp on the reliability of edge cases. And if you want the cheapest software on the planet -- free (as in beer) software -- both development time and quality will suffer as a result.

    In essence, our pool of resources is limited to the available time of the volunteers who maintain that particular project. Therefore, if the quality of a particular module isn't up to your standards, you will need to look at getting that project the resources it needs, and the surest way to do that is to do the work on it yourself and contribute to the community. In terms of the trade-off above, Good trumps Fast and Cheap in your opinion.

    That said, most programmers (myself included) are quite content to use what is already available and live with the limitations. There's nothing wrong with that, it's simply an issue of priorities -- Cheap trumps Good and Fast every time.

    *Or you're working on something that will kill people if any bugs are left in the final product, which if that is the case you shouldn't be using Perl.

Re: Reliable software OR Is CPAN the sacred cow
by Unanimous Monk (Sexton) on Sep 15, 2006 at 19:45 UTC
    I wrote this to discuss another thing: what you think about right attitude, goal and priorities used while developing reusable software like CPAN modules. Do you agree reliability and security must be 1st and 2nd priorities, or not and why?

    If that is truly what your asking, you sure went out of your way to hide it...

    Reliability and security must be 1st and 2nd priority to all CPAN authors? Of coarse not, generalizing any statement like that is folly. Blindly deciding to make reliability and security a priority without taking into account the purpose of a module is going to result in poorer programming in the long run. Each module's priority should be based on its purpose.

    For example, why should the author of Acme-Apace-Werewolf make reliability and security a priority of his module? Technically, it doesn't even need to compile to serve its purpose.

Re: Reliable software OR Is CPAN the sacred cow
by eric256 (Parson) on Sep 15, 2006 at 20:54 UTC

    First Point:Reliability and securtiy mean different things in different contexts to differnt people. Depending on the different modules, no that isn't and probably shouldn't be the first goal. If a module is reliable and secure but unusable what have you gained?

    Second point: Module authors goal match the goals of the project they were developing when they decided to code the module. Which normaly falls in the "get it done" category along side the "works for me" category. These don't mean the module wont be secure and reliable, it simply means that the level of security and reliability will match the goals of the project, not some preset goals for releasing software.

    Final Point: If cpan was started with only "secure" and "reliable" modules with no room for immature modules then you significantly raise the bar of entry. People who would have published modules would not have, they wouldn't have learned and grown an we wouldn't have the better modules they latter produce or the modules inspired by their first attempt. CPAN and perl would be poorer for the loss not richer, at least that's how it looks from here.


    ___________
    Eric Hodges
Re: Reliable software OR Is CPAN the sacred cow
by shmem (Chancellor) on Sep 15, 2006 at 18:58 UTC

    Dear powerman,

    almost all has been said so far in response to your valuable posting, but one point is missing.

    print "Howdy, world!\n";

    is one reason why people code in perl, which is really two, as expressed elsewhere:

    • you can babble in perl, drawl, jargonize, extemporize, speak volumes, drone, singsong, ad-lib, or eloquently express yourself with mastery; and
    • as the saying goes, "perl is the language to get your job done before your boss fires you." Corollary: "you do a good job as long as it's worth doing it well" (something like this is in the first Camel Book).
    Your way of speaking seems like trying to lay down the law.

    Another reason for coding perl is aptly demonstrated by

    $=[!print$=]=!map{$==$=[$#z]+=$=[@z]/@z,($=[@z]%=1+$#z--).=0}@z=@=for@ +==(1)x pop
    which snippet calculates the Euler constant (1st argument is number of digits - credit is due to a wizard on the #perlgolf irc channel (not me :-)).

    Another way goes like this:

    use HTTP::Cache::Transparent (); use LWP::Simple qw(get); use Compress::Zlib (); HTTP::Cache::Transparent::init({BasePath => "/tmp/http-cache-transpare +nt-$>"}); my $gzipped = get "http://www.perl.com/CPAN/misc/lwall-quotes.txt.gz"; my $quotes = Compress::Zlib::memGunzip($gzipped); my @quotes = split /\n%%\n/, $quotes; print $quotes[rand @quotes], "\n";

    Yet another splendid way of writing perl are the gems Erudil has posted.

    A flavour of perl coding which I particularly like and greatly admire is Lingua::Romana::Perligata --

    #! /usr/local/bin/perl -w use Lingua::Romana::Perligata; adnota Illud Cribrum Eratothenis maximum tum val inquementum tum biguttam tum stadium egresso scribe. vestibulo perlegementum da meo maximo . maximum tum novumversum egresso scribe. da II tum maximum conscribementa meis listis. dum damentum nexto listis decapitamentum fac sic lista sic hoc tum nextum recidementum cis vannementa da listis. + next tum biguttam tum stadium tum nextum tum novumversum scribe egresso. cis

    With these examples I'm trying to mark (some) cornerstones of what perl is, and all of these are good uses of perl and splendid ways to make life interesting.

    Ah, and don't let us forget the Acme namespace - that's all more Monty Python than any so-called python (scnr).

    <update why="the point">
    None of these coding examples are in any way meant to be reliable in other sense than "if it ain't broke, don't fix it".
    </update>

    I've upvoted your post, partly because it does adress a valid concern, partly because it rouses a good discussion, and partly because it's good if from time to time a pilgrim comes back to the monastery to shout

    YOU ARE ALL SINNERS! REPENT!
    at the fellow monks sitting at the table. I hope you'll not loose your harsh tone and your engagement in the quest (feel free to set up the perlgems.org domain - it's untaken at the moment - to publish the Carat weight of CPAN modules you come across :-), but don't go down the wrong way, don't end up like Jorge of Burgos who died poisoned by his own wrath.

    Anyways - if you want really reliable code - you probably shouldn't use perl at all. Perl comes with all the ambiguities natural languages have. You should take the right tool for that, C perhaps, or assembler, and scrutinize the libraries your code uses, the underlying OS and the chips that OS runs on.

    Last thing - CPAN isn't a holy cow at all. It's more aptly described as an ecosystem - there are so many interdependencies of modules, which reliably infest each others with parasites, pollution, bugs and even sudden death, cot death... there's no way to stand that but with the firm belief in your own open eyes, skill, and the good intent of your fellow coders.

    Your good intent is highly welcome .

    --shmem

    _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                  /\_¯/(q    /
    ----------------------------  \__(m.====·.(_("always off the crowd"))."·
    ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
      Actually, I'd say that CPAN *is* a Holy Cow...

      As in, "Holy Cow! Look at all that code!"
      Or "Holy Cow! I didn't think of doing it *that* way!"
      Or "Holy Cow! I can actually talk to the person that wrote this!"

      As many have pointed out (and I'm paraphrasing), CPAN isn't some faceless corporation where you are only a consumer of product... CPAN is *real people*.

      And that's a good thing.

      -- WARNING: You are logged into reality as root.
        It's people. CPAN is made out of people. They're making our code out of people. Next thing they'll be breeding us like cattle for code. You've gotta tell them. You've gotta tell them! CPAN is people!
Re: Reliable software OR Is CPAN the sacred cow
by radiantmatrix (Parson) on Sep 15, 2006 at 18:19 UTC

    I think you're missing a huge part of the picture of OSS development, and what CPAN is. CPAN is a way to share code.

    Someone, one day, needed to parse mail messages from an internal system, and wrote a module to do it. It works reasonably well, but doesn't consider all of the edge cases. Still, though, they spent hours just getting what they have to work, so they upload it to CPAN so that no one else will have to duplicate that work.

    Yes, this means that most of CPAN isn't of the highest quality. That doesn't mean that the advice to "go find a CPAN module to do your task" isn't valid: it just means that it won't save you 100% of the work you need to do.

    You claim to have, on your website, better-quality versions of several modules which you felt were incomplete. The point behind CPAN is that it's collaborative: have you sent patches (with tests) to the module maintainers so that other people can benefit from your improvements?

    When we say "don't reinvent the wheel, use a CPAN module", we mean that you shouldn't re-do the work others have done. If no CPAN module quite cuts it, then download the closest one and build on it to complete your solution. Ideally, then, also contribute your improvements back to the CPAN module.

    The responses to your questions about mail parsing were not as you phrase them. They were "don't start from scratch, a lot of work has already been done and is available on CPAN: start there".

    <radiant.matrix>
    A collection of thoughts and links from the minds of geeks
    The Code that can be seen is not the true Code
    I haven't found a problem yet that can't be solved by a well-placed trebuchet
Re: Sacred Cow (was:Reliable software OR ...)
by Intrepid (Deacon) on Sep 16, 2006 at 01:17 UTC

    powerman wrote (Sep 15, 2006):

    Few days ago I've asked about Reliable email parsing and that discussion uncover some points:
    • People doesn't need reliable perl modules, worksforme is enough.
    • People even doesn't understand what reliable solution is good thing which must be first priority goal (even if you can't reach this goal right now)!
    • CPAN == goodness. Use CPAN module - is only recommended solution for anything. If you say all existing CPAN modules for {any task here} is wrong because {any reason here} - you got a lot of '--', no matter is you right or not!

    I downvoted this post -- not the poster -- because it signifies an erroneous mental process; the poster reached incorrect conclusions. I downvote those incorrect conclusions. In brief here's what's incorrect (and btw I've gone through and upvoted the other reply nodes that say the same thing ;-).

    • People at Perlmonks "don't do anything in reply to questions but say use such-and-such CPAN module for that" (paraphrased). (BTW, paraphrasing isn't "putting words in someone's mouth", especially when the original words were barely parseable English to begin with. Nonetheless I will of course welcome and acknowledge corrections from the OP if he feels that I got anything wrong).

      It's a grossly inaccurate generalization, is what's wrong. The very diverse Perlmonks membership comes up with many different sorts of recommendations in response to questions.

    • People at Perlmonks "claim all CPAN modules are reliable solutions". Nowhere. Never. Well, hardly ever. In fact it's a basic and established understanding among knowledgeable Perl people that CPAN contributions vary widely in quality, from the sublime to the absurd. If that informal but widely referred-to (directly or indirectly) understanding never percolated through to the OP, that's too bad. Maybe it does need to be stated more often. But in direct answer to the direct question "Can I rely on it just because it comes from the CPAN?", I've heard no one ever reply "Of course" or anything like it.

    • "People even doesn't understand what reliable solution is" =>
      "People don't even understand what [a] reliable solution is" ...

      Wrong because it is also grossly generalized, but partly true because it is grossly generalized ;-). Perlmonks possess all sorts of degrees of understanding about what quality is, in perl code. Some here have demonstrated the highest awareness of the value of workmanship and pride in one's craft, and some here fall far short of that. Some haven't yet really tried to show what they've got it in them to accomplish, because, perhaps, they fear to fall short of an ideal. In general, though, one thing that characterizes the concerns of people in the Perlmonks community is quality and correctness, far more than you'll find in nearly any other Perl-oriented project or site anywhere.

      I'll just point out that one thing which demonstrates one's own pride of workmanship is the degree to which one makes efforts to create clear and readable documentation for one's own projects, and by extension, to ask questions or comment to others in clear and readable lingua franca of hackerdom (that is, English). A NNES* could ask for someone with better skills to proofread his-or-her writeup before posting. That would set a good example, would show walking-the-walk instead of just talking-the-talk.

      * NNES: Non Native English Speaker

    As for what the OP (powerman) got mostly right:

    • Yes, "they all are low-level modules, each doing small simple task" has considerable truth to it. And that actually makes sense, and is in keeping with the original Unix philosophy of having a rich toolkit of many small tools which is each specialized and optimized for its task (and then connect them together into larger assemblies as needed). Perl came from the Unix world. It's sadly true but also perhaps not unexpected that quite a few modules that work on a higher level, which are on CPAN, are not of such high quality. Some are not maintained -- they workedforher for the original author but that author wasn't fully committed to portability and bug-stomping (which can end up being nearly a full-time job in itself, or seems that way at certain moments).

      I say "ANY software", but especially this important for all reusable things like perl modules and core things [...
      All Perl modules are not "equally reusable". That's a mistaken idea. The basic notion of modularity in Perl is simply the facilitation of extending the language to accomplish different tasks in better ways. "Tasks" in turn are rationally categorizable into "less" or "more" specialized. The more specialized the task, by definition the fewer users need to do it. No?

    There are problems with CPAN and over-reliance on it, I agree. I've seen it grow worse recently. Newer modules are being released which, while not especially low-level (i.e. they are more specialized) are nonetheless of interest to a lot of people, and these modules in turn are relying on prerequisites (dependencies) on CPAN that are failing tests on at least some platforms and aren't being fixed. There's a "quantity over quality" attitudinal problem on the part of some of the most prolific recent CPAN contributors that is degrading the overall reliability of the Perl-CORE+CPAN system and it shows up most negatively where the modules in question pertain to Perl admin and development itself, i.e. building, packaging or installing Perl modules in various ways. These tools need to be the most tested and robust (IMHO) because getting all other CPAN offerings relies on them to be working. I don't know what to do about this trend. Partly I hope it will just correct itself.

        Soren A / somian / perlspinr / Intrepid

    -- 
    Words can be slippery, so consider who speaks as well as what is said; know as much as you can about the total context of the speaker's participation in a forum over time, before deciding that you fully comprehend the intention behind those words. If in doubt, ask for clarification before you 'flame'.
Re: Reliable software OR Is CPAN the sacred cow
by CountZero (Bishop) on Sep 16, 2006 at 08:48 UTC
    Did your version of Perl come with a guarantee that the CPAN-modules would conform to your idea of a code repository? Mine didn't.

    I'm sure there are other programming languages that come with promises of inherent reliability and security ("Eiffel" springs to mind), so why don't you go there and preach your gospel to the converted over there?

    My experience is that the vast majority of the Perl community can work reliably with CPAN, so it becomes a matter of "I'm not wrong the whole world is wrong".

    CountZero

    "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

Re: Reliable software OR Is CPAN the sacred cow
by Booger (Pilgrim) on Sep 15, 2006 at 16:35 UTC
    I probably shouldn't bother replying to this because I know it's just a flame... but ....

    I think you're a foisting idealistic concepts on what is almost an entirely volunteer operation (I say this knowing that there are a few fortunate souls out there who are able to convince their employer to fund development that eventually finds its way to CPAN). I've spent quite a few years working with and for non-profit organizations that depend heavily on volunteers and these experiences have shaped my understanding of volunteer-oriented organizations.

    The vast majority of volunteers give freely of their time and volunteering is only a part of their otherwise busy lives. You can't expect a volunteer to produce the same quality of work as say a professional engineer who is being paid to design a bridge. Sure, some volunteers will produce work equal to or sometimes even better than said engineer but this isn't something you should expect from them. Rather, you should be thankful that they're contributing in the first place!

    Secondly, a volunteer-based organization WOULD NOT EXIST were it not for the many volunteers performing their often thankless jobs. When working with volunteers you must be prepared to work alongside those who are highly skilled, those who are not and all manner of individuals, personalities and talents.

    Most importantly, don't castigate those who give freely of their time. You'll quickly discover that volunteers resent this and will flee from any organization that does. Volunteers aren't thanked enough.

Re: Reliable software OR Is CPAN the sacred cow
by zentara (Archbishop) on Sep 15, 2006 at 16:13 UTC
    CPAN has huge amount of modules for this task, and no one of them support all email-related RFC. ....... AND NO ONE THINK IT'S NEEDED - at least comments to my question say so......... "Yeah, why you think you need reliable email parser, what you will use it for? Forget about it, man, you don't need it. Nobody need it!" :(

    You sound so intelligent and knowledgable on this topic, so why don't YOU write the module for the uber-superior absolutely foolproof email parsing.


    I'm not really a human, but I play one on earth. Cogito ergo sum a bum
Re: Reliable software OR Is CPAN the sacred cow
by Anonymous Monk on Sep 16, 2006 at 00:56 UTC
    Bloody DJB fanboys and their useless rants! You guys pop up every now and then on any programming-related mailing list or forum, spew forth about how nobody understands "secure" programming except for you and the celestial DJB and then vanish again because it takes you six months to code a secure way of opening a file.

    Answer me this, oh powerful one, if your esteemed deity's way of programming is so ultimately superior why does noone (not even the vocal members of the qmail and djbns mailing lists) use the code as he wrote it? They all use extensions and enhancements someone else added (which by the somewhat insane license DJB uses would be illegal if anyone cared) or rewrite the whole shebang themselves because otherwise these apps might have been useful last decade but certainly are not now.

    So yeah, go ahead, program your own POWER::SUPER::IAMGOD:: namespace as long as you like and revel in your superiority. The rest of us will happily go on using modules that actually do the work. Feel free to climb down from your tower whenever you want to contribute to the real world.
    A reply falls below the community's threshold of quality. You may see it by logging in.
A reply falls below the community's threshold of quality. You may see it by logging in.
A reply falls below the community's threshold of quality. You may see it by logging in.