Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Musings: Why do well-intentioned projects go so wrong, so often?

by sundialsvc4 (Abbot)
on Dec 04, 2007 at 13:39 UTC ( #654785=perlmeditation: print w/replies, xml ) Need Help??

There are books about it. Many books. Books about projects that started with good budgets and good intentions, and in the right language and with the right tools, and eight or nine months later they've been to-and-through the last ditch “death march” and they're still not done. Only now you have half as many programmers as you used to. What goes wrong? Why so many books?

I see the same sort of things over and over; we all do. (Hence, the books...) But a few things in particular stick out in my mind. Oh Monks, what sticks out in yours?

Take the time to define “what does it do, and for whom does it do it?”
Two-thirds of the books about Web Pages That Suck talked about the same thing:   if the site does not do what it's supposed to do, for the people for whom it's supposed to do it (inside or outside the company's walls), it's not going to “be successful.” If you don't do that first, it can be the quintessence of Perlishness and still flop.
Beauty is skin-deep
Visual design is not enough, and you can't wait until visual design is perfected before you start building the stuff upon which you will ultimately attach that pretty “skin.” Don't build a blonde. ;-) Don't be beguiled by a pretty but vacuous smile; a bouncing icon; a clever coding trick; a meaningless Flash; cleverness or difficulty.
Build (or steal) the foundation first
“Build,” of course, means “use CPAN shamelessly.” Whatever you're doing, you're not the first one to be doing it. So, don't build anything from scratch. But do build the foundation first, and build well. And build as little as possible. There are dozens of frameworks out there, and they're not all just for blogs. You'll never get a second-chance to do your framework.
Don't repeat. I repeat: don't repeat. ;-)
Start with a town, not a planet
So it's going to do everything in the world someday? Start small. Deliver often. But build (or steal) your foundation well enough for Atlas himself to stand on.
Don't “scrap the old and start over.” Don't even say in public that you might.
That crufty old “legacy” system used to be the greatest thing since sliced bread, and it doesn't take an accountant to know that, in just a very-few years, you or (more likely) your successor will be saying the exact same thing about this one. If the old system was “built wrong,” have you changed and improved your process to be sure that this one won't just follow it down the same old dog-track? If the present system is really not-so-bad, design and build upon it.
We've already got the reputation of being an “engineering” industry that builds things costing millions of dollars which have a business life-span of no more than four or five years. Accountants, CFOs and CEOs hate that. But they love terms like “return on investment” and “durable.”

Your thoughts?

  • Comment on Musings: Why do well-intentioned projects go so wrong, so often?

Replies are listed 'Best First'.
Re: Musings: Why do well-intentioned projects go so wrong, so often?
by chromatic (Archbishop) on Dec 04, 2007 at 16:07 UTC
    Take the time to define “what does it do, and for whom does it do it?”

    I hope you don't mean the hoary old "Plan me harder!" chestnut. isotope and I discussed this the other night, and neither of us can think of a project where completely nailing down requirements before writing code actually made a project succeed.

    Contrarily, we could both think of plenty of projects that almost released on schedule and very nearly came in on budget but never satisfied the users despite stubbornly sticking to the jot and tittle of the original specification. That's okay, though, because next time they'll plan harder!

      I'm with you on this one. My corp is always saying that we need a complete set of requirements to start the work.

      I couldn't disagree more. As soon as you have enough requirements, you should start and then start iterating through solutions. What's "enough"? I'd leave that up to the lead developer.

      I used to drive a Heisenbergmobile, but every time I looked at the speedometer, I got lost.
        My corp is always saying that we need a complete set of requirements to start the work.

        If you had a complete set of requirements in detail sufficient to describe the problem to solve in its entirety, you would already have the complete source code to the software.

      I don't think that's what sundialsvc4 meant. I think he was talking about "knowing the audience" and knowing the range of the project. You do not need to know how exactly will everything look or work, but you need to know what's the purpose and range. And not try to be everything to everyone. But maybe that's what I would like him to think by that :-)

        You got it right.

        The question of “what does it do” is much more important than “how does it mechanically go about doing it.” Most machinery has a protective cover; so does programming. It is a surprisingly easy thing to buy-or-build a beautiful looking machine that does exactly what its designers intended, but that doesn't do what you need it to do. You get the very-uncomfortable feeling, uncomfortable and also very-correct, that the design team never really understood what was needed. So the machine is tight, beautiful, well-made, and utterly useless.

        Another issue can be the unspoken requirements. For example, no matter what my inward- or outward-facing website is supposed to do, I don't want it to be shame-faced by some "133t h4x0r d00d," and I also don't want it to wind up on The Great Grand-Nephew of Web Pages That Suck. I want it to be, as IBM would say, “Reliable, Availabile, Serviceable.” To have “merchantability and fitness to a particular purpose.” That takes a tremendous amount of discipline to achieve as a consistently reliable and predictable process, and most shops don't seem to be able to do it.

Re: Musings: Why do well-intentioned projects go so wrong, so often?
by jhourcle (Prior) on Dec 04, 2007 at 15:19 UTC
    Take the time to define “what does it do, and for whom does it do it?”

    I think there's more to it than this -- it's an issue of managing expectations. and knowing your audience.

    Many programmers aren't willing to roll out without the full functionality, for fear they'll be forever judged by first impressions. (and I admit, I've done it more than a few times) For many larger projects, you need to get some of your intended audience involved early, and have them test incrementally, rather than to tell you that the whole thing sucks 2 weeks (or 2 days) before you're supposed to go live with it. Giving them a chance to interact with the system early and make recommendations can help get their buy-in, and they can champion the project to their peers.

    You need to really know what it is that people want from the system, and be realistic about what you can deliver given the resources available. If there's something that people really want, but won't be available 'til version 2.0, let them know, and let them know early -- you don't need them badmouthing they system because it's missing some (seemingly minor) functionality that they really wanted. If there are things they truly need, then you either need to make sure it gets into the first public release, or only roll it out to a subset of the audience that doesn't require that function. (which means it also has to play well without whatever systems already exist, and hopefully, people won't have to keep switching back and forth between the old and new systems)

    Programmers need to talk to their users -- they know what they want, normally. They might want stuff that we can't give them, but they can let us know what is critical for their work. If we don't understand what the purpose of our systems are in the larger context of their work, we're doomed to fail.

      I would dare to reword and redirect the last paragraph:

      Programmers need to be allowed to talk to their users -- they know what they want, normally. The users might want stuff that the programmers can't give them, but the users can best let the programmers know what is critical for their work. If the programmers don't understand what the purpose of their systems are in the larger context of the users' work, we are all doomed to fail. "Send your questions to our project manager, who will tell it to the secretary, that'll mail it using her own words to the secretary of the (whatever level) boss within the client, who'll eventually forward it to someone who may eventually end up using the system being built and whose response wil travel the same way and end up in the developers mailbox within mere two or three days" doesn't cut it. Don't protect the developers from the potential users and the potential users from the developers!

        Just to reemphasize your rewording and redirection, the delays and paraphrasing that you mention are just the tip of that iceberg. In my experience, lack of direct contact with the actual end users tends to end up with the software being built to do what the end users' manager thinks they need (or, still worse, what the manager thinks they should need), which often has little or no connection to the reality of what the end users will be doing with it.

        Design-by-management-proxy also tends to lead to the software being created solely to duplicate the existing process(es), but with newer, flashier technology, while allowing the developers to get in there, talk to the users, and actually gain at least a rudimentary understanding of not only what the end users are/will be doing, but also why will often allow for better processes to be developed.

        I know where you're coming from, but I think we're in two slightly different situations here ... you're likely dealing with a case where people are forced to use the project. I've been in that situation, and people complain about what they don't like, but they might not send it to the right people, and might take days for it to reach the developers (if ever).

        On my more current work, I'm building software that is completely optional. It's intended to help people, but no one's being forced to use it. When I built the system for Fark -- if people weren't happy, they'd leave ... we had no idea why, and really, we didn't care. (this was well before Drew was selling advertising) The submission queue was just to make it easier on the admin, so we could weed out the crap people were submitting.

        Now, I'm working on software to help scientists find data. They have other ways to find their data, so most don't bother using what we have ... if they use it, and aren't instantly happy with it, they don't fill out the feedback forms or e-mail us ... they just go away. So, once or twice a year, I go to conferences to tell people about new features and try to get the scientists to tell me what they want as new features ... some aren't willing to tell you what they're looking for, as it might tip their hand on what sort of research they're trying to get funding for, others are just happy with their current methods and don't care.


        Of course, in some ways, I've probably harmed the users by taking to them. I tried sending out an explanation of what's wrong with some of the catalogs, and someone asked 'what's "normalized data"'? We speak completely different languages when talking about the same problems, but I can't design an interface for their use without understanding how they think about their problems and the words they use to discuss concepts within their specialty.

        But, anyway, I think jenda and I both agree -- developers and the users need to interact, so we can understand what the users want and can fix the problems they find.

Re: Musings: Why do well-intentioned projects go so wrong, so often?
by KurtSchwind (Chaplain) on Dec 04, 2007 at 16:17 UTC

    Big topic. Why do projects fail? They fail on expectations.

    You have the expectations of the intended audience. What do they think they'll get? Often we set them up for failure because we ask them to come up with a pie-in-the-sky laundry list of wants and say "uh yeah, we can do that." And we aren't lying. We can. But they probably won't get it all on first release.

    And then there are the expectations of the development crew. How much testing? How much can we count on the QA group? How much can we count on the user group? How much input do we get in how we solve problems?

    It's more amazing that large projects work, than that they fail. And it takes a special skill set to pull it all together and keep things going. Your ace number 1 perl guru probably doesn't also have that skill. I'm not saying that the project manager should get paid more than the ace guru, but I am saying that you have to respect that it's a different skill set and it's a pretty darn important one.

    I used to drive a Heisenbergmobile, but every time I looked at the speedometer, I got lost.
Re: Musings: Why do well-intentioned projects go so wrong, so often?
by Mutant (Priest) on Dec 04, 2007 at 17:03 UTC
    Don't “scrap the old and start over.” Don't even say in public that you might.

    I think you've chosen a solution without defining the problem here (or at least, you've got a specific solution to a very general problem). There are plenty of times where it's a good idea to scrap everything and start again. This can be from the technical end (e.g. the code base is barely maintainable, the design has fundamental flaws, the infrastructure is no good), or from the customer end (e.g. the system never worked properly, or did what we wanted it to do, the business has moved on, numerous "paradigm shifts" have occured).

    I agree, it's not a good idea to scrap everything without a good reason. But good reasons often crop up. And it's important that the "business" type people, who are often the ones who write the cheques, recoginise that this might not be something initiated by them. Just because something "works fine" now, doesn't mean it's economical to leave as is, when every minor change takes weeks to complete.

    More generally, I think Agile can offer a lot of help in most of these issues. I prefer Scrum with XP-style development practices. A lot of people still baulk at the mere term "Agile", but a lot of that is because it's been adopted as a buzz word, and completely mis-interpreted. Even if you do understand the theory side, putting it into practice can be a lot harder than you might think, especially in large organisations, with a lot of "baggage" entrenched in both processes, and people's expecatations. (Hint: to put it into place in this sort of environment, you almost *have* to have an independant Agile Coach. You'll probably find that as soon as you start trying to remove impediments, you'll find huge webs of resistance from upper management that are difficult to break down yourself).

    I think what people who've put Scrum in to practice well find, is that if they really want to produce working software on a regular basis, they have to address how the entire organisation operates. To me, that's a reflection on the fact that software has become so integral to so many business, and is much more complex than your average (non-development) project.

    Bottom line is: getting it right is hard. And we're still learning how to do it.

    As a side note, it probably goes without saying, but there is only so much you can learn out of books. Experience is crucial, including learning from others' experience.

      My brain generally clicks-off when people drag out sports-analogies for any reason. Programming isn't a sports-event; it's engineering. (Let the record show, furthermore, that “armchair quarterbacks” actually have no idea how a professional sports team, nor the surrounding business-entity, actually works. They drink the beer, maybe pay $34 for the book, and imagine that they do, getting smarter and smarter as the afternoon wears on.)

        I'm guessing you're referring to the name "Scrum"? Well, it was actually named by a couple of Japanese guys who clearly didn't understand Rugby, because their analogy doesn't make any sense. But at any rate, it's just a name, and the methodology *is* based on engineering (specifically, lean manufacturing).

        I suppose they fancied themselves as marketing gurus. The fact that they originally named the Scrum Master role "Hooker" (a position in Rugby) proves they certainly weren't.

        Anyway, I'd strongly recommend you go past the name and at least *look* at Scrum (or other Agile methodolgies). Even if you don't want to use it, you may find parts of it interesting and valuable.

Re: Musings: Why do well-intentioned projects go so wrong, so often?
by starX (Chaplain) on Dec 04, 2007 at 20:20 UTC
    I agree, most especially with taking the time to define what you're doing, and designing the foundation first. I think a better way to say "start with a town" is to focus on the details of the components when the time is right.

    Good design, whether it's for a program or a building, needs to begin with a conceptualization phase, which in turn leads designers to figure out the best way to implement their designs. All too often people want to jump right in and get their hands dirty with some code, but this has a tendency to make more work.

    I've found that too much planning can be as bad as not enough, though. Too many people sitting around not actually doing anything for too long saps energy and time. It's always best to begin the design process with a simple overview of what you're trying to achieve, and this is where a good manager is critical, keep everyone focused on that.

Re: Musings: Why do well-intentioned projects go so wrong, so often?
by Tabari (Monk) on Dec 05, 2007 at 09:39 UTC
    Don't repeat. I repeat: don't repeat.

    I couldn't agree more. Why, by the way, buy books when so many interesting experiences are collected in such a small space.
    Anyhow,I have seen the above rule being applied to the team too, which was not a good idea. Try to learn from the past by mixing up teams having different technology skills and different functional skills.
    Build to Last (ISBN 0060566108) was an eye opener from an organizational point of view. A development department is no exception.

Re: Musings: Why do well-intentioned projects go so wrong, so often?
by apl (Monsignor) on Dec 05, 2007 at 16:16 UTC
    One word: Management.

    I've lost track of the number of times I've informed management where a project was and when it would be ready to be released, and they've responded by telling me when they wanted it released (usually one to two weeks earlier).

    How difficult is it to understand "We need to regression test before we can release?"
      Regression Testing?!?! Bite your tongue!! Why would Management want to discover regression issues? "Those are issues that 'Rev. 9 million 5' can fix"..."an undocumented feature"!!! :-D

      I agree with apl we must have Regression Tests, you must discover what broke when you changed/added 'XYZ'!!! No excuses for not doing Regression Testing, however Micro$haft ships patches without testing all the time...forcing every IT shop in the country to do their OWN testing before deploying a fix enterprise-wide.


      "No trees were harmed in the creation of this node. However, a rather large number of electrons were somewhat inconvenienced."

        One doesn't have to dwell on Regression Testing. Management has their priorities even as Programmers do. The difference is, they don't usually tell us why they need to shave time off the estimate. All we see is that we're being forced to ship something which we know is not complete.

        But who gets the blame? You'll notice the title of this thread is not Who do so many projects get mismanaged, so often?.
Re: Musings: Why do well-intentioned projects go so wrong, so often?
by graq (Curate) on Dec 05, 2007 at 08:13 UTC

    Theory begins by denying reality - to talk about reality, to go around reality, to catch anything that attracts our sense - intellect and abstract it away from reality itself.

    Begginning with saying that the outside world is not a fact, that existence can be doubted and that every proposition in which reality of the outside world is affirmed is not an evident proposition but one that needs to be divided, dissected, and analysed.
    It is to stand consciously aside and try to square a circle.

    Projects fail on expectations and bad criteria. Everyone gets them mixed up and start talking about who's methodoligies are better.

    Perhaps, one day, after one has delivered a project where, for example:

    a) Scope and functionality is to be reduced to fit the deadline
    b) Performance is not an issue
    c) Content accuracy and reliability is expected to be wrong at least 6 months after deadline

    That is when one should start discussing achievability, business projections and the stock market.

    Like an ancient buddhist monk, you should transcend all your teachings and simply deliver the solution. Everything before it and after if is the journey of your life.

    (Apologies for my badly recollected quotation)

    -=( Graq )=-

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://654785]
Approved by marto
Front-paged by Arunbear
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (3)
As of 2022-10-05 03:16 GMT
Find Nodes?
    Voting Booth?
    My preferred way to holiday/vacation is:

    Results (20 votes). Check out past polls.