Thank you. I had never heard of the ExtremePerl book and now that I know, at first sight it already seems to be a precious resource (for free)!

-Andrew.

You could also try an iterative development model as an improvement over the basic waterfall model. Its basically a whole bunch of little waterfalls that incorporate their own requirements - design - build - test iterations before a client review that is fed into the next iteration.

You start by identifying the high risk issues that are likely to cause problems and tackle these areas first. Any severe problems can be dealt with before too much money has been spent on the project. Reporting on successfully testing the difficult parts of the project will build confidence with the project manager and the client because that's what they worry about.

I second tomazos' view that a prototype (even of the smoke and mirrors variety) is a great way to start the design process. Reviewing the current product (demos, models etc.) with the client at each review stage allows them to better visualise the deliverables and provide better requirements which are fed into the next iteration.

I respectfully disagree. The theory is sound; the practice is not.

Prototypes tend to become products without further design; no one wants to revisit the work done, and if the design isn't done right to begin with, it's usually not done right at all.

Simple little programs that "evolve" tend to grow into sprawling monstrosities who's fundamental design assumptions are often wrong for the scale of complexity the end product is expected to run at.

For example, on my desk I have a paper that presents a formal proof that certain aspects of concurrency have to be designed into a system in the first place; any attempt to add them as an afterthough is guaranteed to fail in certain important aspects.

Sometimes, the "reduce the program and then re-expand it as needed" philosophy fails utterly. As the saying goes: "You can't jump a chasm in two short leaps." Sometimes, it's wise to take the fundamental achitectural considerations into account in the first place: if you know you're building in mountains, plan to build a few bridges. If you're working below sea level, don't assume tunneling is a good idea. And so forth. Don't abstract away the important implementation details, because they're what will make your product sink or swim in the real, non-abstract, non-ideal world.

I don't like prototypes; they're a good indication that the client doesn't really know exactly what they want. That's a red flag for any project, of any sort, right there. Prototypes tend to drag troublesome people out of the woodwork with all sorts of mutually-contradictory (and self-contradictory) opinions on how you should design the system, who will then remain eternally bitter that you asked for their advice and didn't take it.

Sometimes it's better to just work out what really needs doing, and do that. Here's an all too typical scenario:

Boss: "Bill in marketing needs a TPS report sent in with blue headings on a daily basis ASAP! How long will it take you to complete?!?" Me: "I'll find out, and give you an estimate by morning."

I go see Bill... Me: "Bill, why do you want blue headings on the TPS reports?" Bill: "Mary wants the entries will blue headings sent to Accounting for their summary report, and the entries with the black headings sent to Payroll" Me: "Thanks"

I go see Payroll... Me: "Why do you need the TPS reports separated out?" Payroll: "We don't really use marketing's data; we thought Mary needed this change made! Why don't you ask her?"

I go to see Mary... Me: "Hi, Mary. I'd like some more info on the TPS report Blue Heading's project" Mary: "Oh, well, I'm not a very technical person. You should talk to Fred... he's handling the matter for me"

I go see Fred in Accounting.... Me: "Fred, how do you create the summary reports?" Fred: We take the info from marketing, give it to Joe, and he writes up the summary report." Me: "Is that all you do with them?" Fred: "Yes, that's it. Why?" Me: "No reason. Thanks, Fred!"

I go see Joe... Me: "Joe, what do you do with the info from marketing for summary reports?" Joe: "Marketing data isn't part of the Summary Reports. I just need it to put the total volume from each department into the Department reports, and email it off to Ralph"

I go to HR... Me: "Who's Ralph?" HR: "Ralph? He quit three years ago..."

I go to the boss.

Me: "Job's done." Boss: "That was fast!" Me: "It was easy, once I eliminated all the redundancies..."

I suggest you might enjoy reading Agile and Iterative Development: A managers guide by Craig Larman. It's a survey of several agile methodologies, but what really makes the book a wonderful read is a great section up front that charts the history of "waterfall" development and its many failures, with tons of statistics and anecdotes.

For example, waterfall development really got enshrined as a "best" practice by the US Department of Defense and was picked up by other companies and governments. But the US DOD has subsequently tossed that out and decided that development must be iterative and evolutionary, meaning, among other things, that the design cannot be fully known in advance -- but that switch hasn't been picked up as widely.

I think it's entertaining and thought provoking even for non-managers and gives good ammunition for a more experimental, evolutionary approach to software development.

If I'm developing something for a customer, before I start coding I try to define a rigorous spec that a) the customer can understand as defining their requirement, and b) contains enough detail to ensure the requirement is exactly specified. This can be a challenge with completely non-technical customers.

When coding less formally (writing support utilities when working under contract, writing my own modules etc) I tend to work out a fairly precise but very basic spec beforehand, write the first version; then immediately think of and start adding lots of new functionality.

I break the task down, often something like:

get_input();
process_input();
output();
[download]

The same process maybe repeated with each task depending on complexity or if I'm stuck. In these cases there can be a string of scripts testing quite short subs.

When there is reasonable confidence in the parts it is a case of fitting everything together.

This way I can convince myself that I have some working code with only a small bit that doesn't work rather than a lot of code that doesn't work at all. :-)

It's entirely conincidental that when you're _really_ stuck you have a shortish piece of code, some test input data and, perhaps, an example of the output you need. Very handy if you ask the monks for help!

I have some arbitary rules of thumb:

• Everything must fit on a screen. Subs, loops, if/else blocks.
• Use subs to avoid deep indenting rolling across the screen.
• No hard coded data - at all. Use a config file.
• Use a look up table even if most of data is the same or undef.<.li>
• Don't sweat over arbitary rules of thumb

Update

Oh yes, if there is an and an or and a not on the same line the code is just plain wrong. Rewrite it because you'll (me, at least) never ever get it right. :-)

Some people strongly agree with you on that point. "Plan to throw one away, you will anyway" is one of Brook's rules in the classic Mythical Man Month.

You have independent verification of this truism.

Besides, prototypes are good ways of showing the boss what you've accomplished that day. :-)

What I tend to do w/ code is to prototype something - especially if it's hard either 'cause it's large or it's complex. If it's not something I can fully churn out in a few hours, I do a bit of it, and walk away at a good stopping point. Why? 'cause that is a one time measurement. If I can come back, and see some sense to it, and not want to scrap it, that is my second measurement. More likely, though not completely 100% all the time true, I've done something right. It's like proof reading.

It's the same thing you do w/ client facing interfaces. You propose it then prototype it to make sure that things are really what they want. In your case, you are your client, so you need to do a little, and come back fresh to make sure you are making sense. Taking breaks helps! :)

In my lifetime I've had to plan, design, and produce everything from machines for manufacturing lines to residential hvac systems to decks to musical scores...oh, and computer programs too.

And your observation about this question being applicable to many areas of creative endeavor is very true.

In every case there is an optimal medium to achieve. At some point planning, calculating, goes beyond the accuracy required by the job and you are wasting time. At the same time, diving head first without a minimum of planning leaves you open to pitfalls.

You can buy very complicated heating ventilation design programs. After you spend a week getting the dimensions and r-values of everything in your house, then another week inputting the information, it will tell you what diameter your heating ducts should be for ideal air flow. In the meantime, an experienced hvac guy measures the length of runs, and sketches the main trunks and branches. Counts a rough number of elbows needed. Then, with that reasonable planning done, goes to the hardware store to buy the ducts, the elbows, and a few boxes of screws and duct tape. He'll return what he has extra. He's done while you're still measuring, and the difference in system performance is insignificant.

At the other extreme, the guy down the road set to building his new house with no planning (really). All went well until he discovered the new by-laws on septic systems and his half-finished house sat there for a year while he figured out how he would achieve compliance with property-line set-backs. A minimum of planning would have spared him this major pitfall.

General rule - do a reasonable amount of planning. Of course, what is reasonable is very subjective, and that's where experience and skill comes into any profession. Experience and skill at estimating what is a reasonable amount of planning comes with, you guessed it, experience - that is, mistakes.