Warning: I am very much a proponent of the "bottom-up" method of designing.

Obviously it depends very much on what you're writing, methodologies will tend to vary between a 5-line script to rename your mp3s and a framework to take over and run the worlds stock exchange markets ;-). Nevertheless, here's a rough guide I tend to follow:

1. Create a version control repository or a subdirectory of an existing repository. All through the following steps, commit to that repository whenever you have written more code than you can recite without looking it up.
2. Start out with an essential feature your code is supposed to have which can be easily implemented.
3. Write a test to see whether your code performs correctly. Yep, this test will fail. You haven't written any code yet. Any test that passes immediately after it is written was written too late (sometimes unavoidably so) IMO.
4. Write code which passes the test.
5. Look at the code. Does it not conform to your expectations? Can it be written more cleanly? Can you remove anything? Refactor and make sure the test still passes. Refactor mercilessly.
6. Return to 2 and repeat.

This is more or less completely ripped off from the Test driven deveolpment and Extreme programming methodologies. Books on the subject abound and I very much recommend reading some of them.

If you're serious about it, you can even write tests first.

Anyway, tests can help you very much while refactoring, and in my experience they are 100% worth the time you invested to write them.

Very interesting. Tests are definately an area where I need to educate myself. I expect it is a difficult process to incorporate until you see the benefits.

One method I have found useful is as the "script become larger and larger" I make them smaller. :-) What I mean is to break the script up into subs that do the detail work leaving the main script quite short. If you choose meaningful names for your subs it helps you 'see' what the app is about (I quite like to get that part fit on the screen). Write your subs so that everything it needs is passed in as an argument, perhaps a hash ref, with no reliance on any globals. It is then possible to write a 'demonstrator' script with just that sub in it. Pass in various permutations of arguments and study what it returns.

I like subs that fit on a screen too and sometimes they have little helper subs and this also helps to see what is going on. You can write your 'main' script straight away (a flow chart, if you will) and write and test each sub one at a time.

When you find your subs can be logically grouped together, along with their helpers, there may be a case for grouping them together in a module. It may be that having the module represent itself as an object with methods and properties would be helpful, "look mom! I've got an API!". With some careful planning you could find it reusable. Get a few well organised modules and your script is back to 5 lines again!

What I'm finding is if that is how you've developed your app it is very ameanable to testing, in fact it could be argued that the method I've described is just an inefficient approach to testing. I have lots of tests, why not organise them? We'll see how I get on. :-)

Before your app gets too large be sure to have a thorough look at CPAN. Are you sure no one has trod this path before? CPAN is often the best and quickest way to cut _your_ app down to size cuz someone else has already done the hard work.

My approach hasn't changed much in other respects though. Absolutely nothing gets done until I start writing code. It's what I like doing, all suggestions to do otherwise are cheerfully ignored.

Good luck, and start coding! :-)

I have been using an approach similar, breaking tasks down into chunks (subs) and also writing the interface to a module before coding the module. Yes, I am also trawling through CPAN so I don't reinvent the wheel. These things I feel are working reasonably well for me.

However.....I'm still finding that halfway through the coding I'm either having that 'lightbulb' moment and scraping half the code and starting again or , as I mentioned in my original post, I code myself into a corner...usually with unwieldy data structures that get more out of hand as I progress.

Maybe I need to 'play' with my sub templates a bit longer before I start fleshing them out with the actual code?

In essence, by repeated prototyping you move closer and closer to an approximation of the real world problem. It helps uncover what is "unknown" about the problem and is a great tool for helping an end user (typically untrained in systems thinking) work with you to reach a deeper understanding of the problem they are asking you to solve. In programmer terms, the use of strictly limited timeboxes and regular reviews with the business user ensures you don't wander too far off course.

It contrasts with the historical waterfall method of top down development popular in the 1980s, which is based on the belief you can fully specify the domain of a problem from Justification Study to Feasibility Study to Requirements Analysis to Architectural Design and then down to detailed program spec.

It really depends on the kind of system you are writing. For a system worked on by 500 developers, you're probably better off with a top-heavy PRINCE 2 project management and top-down design approach. Of course, this has led to some classic disasters in developing large scale government systems, for example the DSS client-server replacement system (£500M wasted). I think that's more a function of the complexity of the problem and the limitations of how much one human brain can understand rather than any inherent flaw in the methodology.

For smaller systems, or ones with unclear requirements, a bottom-up approach is better provided someone (the tech project manager) has the discipline to ensure documentation is completed properly and an adaptive approach is taken.

Good luck!

This article is worth a read: Ten Essential Development Practices by Damian Conway.