karmacide has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks!

Not a perl specific question per se, but you lot were the first people that sprang to mind when I realised I needed help. :)

I am about to start writing an improved web interface for the EMBOSS suite for my masters. It will involve perl CGI, perl_mod, replacing some C++ in EMBOSS, and using a large SQL database. The problem is that although I have been programming for a while, this is the largest and most important project I have tackled so far (and I have three months to complete it). I have a good idea in mind for the general solution to the problem. But I was wondering if anyone had good advice for how best to go about designing and managing something like this; hints, tips, or favourite methodologies please!

Cheers to all,

Matt

update: As this is my first post, I really want to thank everyone who replied. Especially as this isn't relly a PERL question. Nevertheless your comments have been both thoughtful and insightful. Thankyou, I will definitely follow your advice.

Replies are listed 'Best First'.
Re: Developing larger applications.
by kragen (Sexton) on Jun 05, 2004 at 02:51 UTC
    1. With source control, such as CVS (although there are alternatives.)
    2. With good design, which is a balance between abstraction (which has many different aspects and ways to describe it, such as "don't repeat yourself", "once and only once", "encapsulation", "abstraction", modularity, and "information hiding") and simplicity. Finding the right balance is a delicate matter of taste. Finding abstractions that allow you to achieve separation of concerns often requires considerable inspiration and experience; the Design Patterns movement is intended to make some of that experience teachable. Patterns won't give you the good taste to know when not to abstract, though.
    3. With automated tests. These can be even better if you write them before the code.
    4. Incrementally, using refactoring instead of rewriting whenever possible.
    5. With mentors. I find pair programming an especially effective method of mentoring, although it won't turn a bad mentor into a good one.
    6. With someone else carefully proofreading your code to find bugs that are too subtle for the automated tests.
    7. By building one to throw away --- perhaps not the whole system, but each piece you're uncertain of the design of. Hack together something quick to test each design possibility, see how it works, then toss it. When you've explored the design space, write the real thing.
    8. Spending as much time reading about how other people have solved problems similar to yours as you spend solving the problems themselves.
    9. With rapid feedback. Having your test fail one second after you write the bug is better than having your test fail ten seconds after you write the bug; ten seconds is better than a minute; a minute is better than ten minutes; ten minutes is better than an hour; an hour is better than a day; a day is better than a week; and so on without limit. Sometimes it's hard to get feedback on whether you've, say, broken your database schema, your web server config file, or your build scripts, within one second or even one hour. Try very hard to reduce that time.
    10. With reliable feedback. An automated test that gives you the right answer 100% of the time is worth a lot more than a test that gives you the right answer 50% of the time. A lot more than twice as much, in fact.
    11. With as much access to the people who will use the code as you can manage. It sucks to spend an hour or a day working on a feature that the actual end user could tell you is useless in less than a minute.
    12. With other people having as little access to you as you can manage.

    I tried to put these in order of importance, but the precedence gets a little fuzzy around #6.

Re: Developing larger applications.
by chromatic (Archbishop) on Jun 05, 2004 at 00:58 UTC

    I like what Paul Graham said:

    I try to build big programs out of small ones. So when I approach a new project, I look for the subset of the problem that I can solve with the smallest program. Then I start adding things.

    I'd break the problem into small features and plan to implement each feature in order of importance. (If you can convince your manager to put them in order, that's even better!)

    The one trick is figuring out what to do first. If you've broken the job up into smaller pieces, choose the minimum you can do that represent the whole system. This'll let you explore the system as a whole as early as possible so you can see what's feasable and working.

Re: Developing larger applications.
by Zaxo (Archbishop) on Jun 05, 2004 at 01:31 UTC

    Tests. Documentation and tests. Before you code each day, write, for what you plan to do, the user-level pod from the requirements for and write tests to verify each thing you say in it. Also write tests for your functions' behavior with context, unexpected input, and system errors. Then start coding.

    After Compline,
    Zaxo

Re: Developing larger applications.
by andyf (Pilgrim) on Jun 05, 2004 at 00:44 UTC
    CVS - even if you do nothing else make sure you CVS.
    Andy
Re: Developing larger applications.
by hsinclai (Deacon) on Jun 05, 2004 at 01:36 UTC
    Validate input.

    Fail securely..

Re: Developing larger applications.
by qq (Hermit) on Jun 05, 2004 at 21:05 UTC

    Source control. But if you are starting from scratch forget CVS and use subversion or arch.

    Try to get, and keep something working. If 3 months arrives and you have nothing to show you're screwed. If it works, but lacks all the planned features (even important ones) you have a chance to explain.

    3 months is not very long, particularly if you have other responsibilities.*

    ... insert all advice from above posts here ...

    qq

    * I read, I believe on perlmonks, some sage advice about how best to respond when the PHB asks if you can 'just quickly do ...'. The best answer is 'Of course, what would you like me to postpone?'

Re: Developing larger applications.
by adamk (Chaplain) on Jun 06, 2004 at 09:02 UTC
    One perl-specific thing I find with big jobs is that because perl coding styles can vary, it can be very useful to have a general style guide for the project. Here is an example to get you started that I've been accumulating over several years specifically for very large perl code trees.

    In general, comment more and use simpler code than you might if it's a small application.

    Some big projects take a while before enough pieces come together to show a snazzy demo system. If you go with CVS, perhaps something ( blatant promo :) to keep management comfortable work is actually moving forwards.

    Test lots, and test first if you can, but if you get REALLY stuck, and can't think of a good way to test something, don't let it stop you or divert weeks rigging complicated emulation systems, at least not at first. Consider a small example or demo script which, while not testing formally, will allow you to run some tests runs on a particular subsystem, and just debug it using the perl debugger.

    If you go the mod_perl route, make as many pieces testable WITHOUT having to run it inside mod_perl as possible. perl -d test_script.pl is an easy way to find bugs. Doing the same on a mod_perl module is often harder without spending large amounts of time writing complicated emulation code.

    Also, write down and write up. By this I mean, write a sufficiently thorough framework to handle the entire system early, and if you find some task or subsystem that can be completely seperated from the rest of the system, stop and get that written and completed first.

    You'll have a nice big framework for everything to fit at the top, and a toolkit of standalone modules/widgets at the bottom, and work your way towards the centre.
      In general, comment more and use simpler code than you might if it's a small application.

      Seconded. Although I would personally emphasise simple code over more comments.

      I often find comments get in the way of simplifying the code. If I write a bit of opaque code and my first response is to write a comment to explain it then the code stays complicated.

      Instead I find it a useful mental exercise to ask myself 'Is there any way I can make this code easier to understand?' whenever I am tempted to write a comment. I usually find that there is and the comment proves unnecessary.

      If you go the mod_perl route, make as many pieces testable WITHOUT having to run it inside mod_perl as possible.

      Not that I disagree with this excellent advice but...

      Doing the same on a mod_perl module is often harder without spending large amounts of time writing complicated emulation code.

      ... Apache::Test has made testing mod_perl code a lot easier recently.

      Also, write down and write up. By this I mean, write a sufficiently thorough framework to handle the entire system early, and if you find some task or subsystem that can be completely separated from the rest of the system, stop and get that written and completed first.

      Personally I have moved away from trying to design and build frameworks up-front. For two reasons:

      1. I find that that my initial design decisions / assumptions prove to be incorrect in the long term. Requirements change. Assumptions prove incorrect. Complex areas turn out to be simple. Simple areas turn out to be complex. If you do too much of the framework design up front you end up throwing it away or shoe-horning code into it.
      2. Working with a more complex framework early in development can actually slow you down because you don't need the extra bells-n-whistles until later in the project.

      Instead I incrementally build the framework as I go along. That way I can keep it flexible in the face of changing requirements, and I don't have to carry any extra infrastructure weight until I actually need it.

      It needs more discipline to keep everything refactored as you go along, but I've found it a more effective technique myself.