http://qs1969.pair.com?node_id=146303

d_i_r_t_y has asked for the wisdom of the Perl Monks concerning the following question:

we've just migrated our 80K line pure perl web application to mod_perl...ah... so much faster... can anyone advise on their experiences for setting up apache/mod_perl for team development? up till now, we've all been running our own copy of sources out of our home directories, and running a separate apache instance for each developer seems like overkill -- is there a better way???

cheers all,
d_i_r_t_y

Replies are listed 'Best First'.
Re: Team development on mod_perl + apache
by lachoy (Parson) on Feb 19, 2002 at 03:55 UTC

    Why does running one apache/mod_perl server per developer seem like overkill? Trying to sort out the issues involved with running all the modules under one server, particularly multiple @INC paths required for different module versions, is extremely frustrating. And the developers will soon be at each other's throats since they'll have to restart the server all the time, at different times. Since you're in development you can keep the number of servers run per developer very low (2-3) so you shouldn't have resource issues.

    It might also be useful (depending on who your users are) to have a single set of lightweight Apache + mod_proxy servers sitting in front of the development servers. Then you can do some fun tricks with mod_rewrite so that a request for http://dev.myco.com/lachoy/ gets proxied (or even just redirected) back to my server running at whatever port is allocated to me. The mod_perl guide has oodles of info on this sort of setup. (A nice side-effect of this is that ordinary users can stop/restart servers running on port > 1024 as often as they like.)

    Chris
    M-x auto-bs-mode

Re: Team development on mod_perl + apache
by trs80 (Priest) on Feb 19, 2002 at 04:18 UTC
    There are many factors that will effect your ideal development environment. Such as:
    • Number of developers
    • Turn over rate
    • Expansion rate
    • Typical developer comfort level working with their own copy of mod_perl/Apache
    • Testing methods
    • Whether mod_perl is needed for those testing methods
    • How each developer interacts with the source
    • How your code is broken up
    • Network latency
    • Network stability

    If you want to operate in a single mod_perl server environment I recommend you implement and promote thorough unit testing and a human gatekeeper that controls/logs all the updates to the main server in addition to CVS or other revision control systems.

    UPDATE Add two items to the list. When I was working with development teams that were on less then reliable networks it became more critical for each development team segment to have their own apache/mod_perl to test on.
Re: Team development on mod_perl + apache
by mortis (Pilgrim) on Feb 19, 2002 at 05:09 UTC
    One of the main issues that drove us to run private instances was the simple goal of allowing each developer to continue working even if an individual developer introduced a compile error into their local codebase. If a module used by the majority of the site was being worked on, and a syntax, compile, or run-time error (like calling a typo'd method name) was introduced, it basicly disabled the other developers.

    We re-organized the source code into a hierarchical directory structure (like you would see in lib/perl5/*) based on package names. We then just added the source directory to the @INC used by mod_perl. Each developer used their own working copy of the CVS archive, something like $HOME/projects/webSite/lib. The document root for the web site, as well as the Perl lib path were both under the same project in CVS.

    This also simplified the release process for us, which basicly ended up being a cvs update on the QA, and then ultimately production servers.

    Kyle

Re: Team development on mod_perl + apache
by maverick (Curate) on Feb 19, 2002 at 07:02 UTC
    Everybody running their own lightweight copy of Apache on separate ports would be ok. You'll either have to have the same lib/perl directory for all the copies, or you'll have to watch out for syncronization problems where two developers change the same module. CVS or some other revision control system is going to be a MUST to keep each copy of the modules in sync if you go the separate directory route.

    On a different note, the app server framework I developed supports dynamic loading of modules when they change (not going through Apache::Registry), thus eliminating the restart problem. And if a certain module chokes, it doesn't bring the server down (you just get an error message when you try to access it). The mechanism isn't that complicated, I can show you if you like.

    Update Fixed grammer errors :)

    /\/\averick
    perl -l -e "eval pack('h*','072796e6470272f2c5f2c5166756279636b672');"

      Yeah I would recommend everyone running on their own port, but keep everyone's development environment completely separate (even lib), and make sure all dev code is controlled by CVS.

      If the developers talk to each other enough they'll be able to work out any weirdness possibly resulting from changes to lib (hopefully anyone changing lib isn't going to hack-and-slash their way thru it ...)

      -mr.dunstan
        ++
        One nice side effect of everyone using and building their own web server is that every one of the developers will know how to get the site up and running, so if things go wrong, any one of them can help out as opposed to the usual shrugging of shoulders and pointing of fingers.
        ()-()
         \"/
          `
      On a different note, the app server framework I developed supports dynamic loading of modules when they change (not going through Apache::Registry), thus eliminating the restart problem.

      Why not just use Apache::Reload or Apache::StatINC? They accomplish exactly the same thing as you describe above. A:Reload also allows selective reloading of modules IIC. But they also have the advantage of having been around a while & well tested. Is there something special your module does? I love to reinvent the wheel myself, but sometimes it's better not to do so. :-)

Re (tilly) 1: Team development on mod_perl + apache
by tilly (Archbishop) on Feb 19, 2002 at 06:42 UTC
    I was thinking about exactly this problem when I wrote Versioned modules. Unfortunately that wasn't the direction that that project wound up going in so I have never really pushed that code.

    I have no interest in maintaining said module. If someone wants to take it, try it out, and load to CPAN, you have my blessing. (Probably needs a new name, a better decide_version, the ability to handle different @INCs for different developers, that kind of thing.)

Re: Team development on mod_perl + apache
by Starky (Chaplain) on Feb 20, 2002 at 09:53 UTC
    I will second and third many of the above responses: Having each developer with an Apache instance on their workstation is not a bad thing at all.

    When major code changes need to be made that will result in things being broken for a substantial period of time, it saves everyone a good deal of heartache.

    In fact, I consider it one of the major advantages of using open source. If you were developing in Cold Fusion, you'd have to spend thousands of dollars a seat to give everyone a development environment they can wreak utter havoc on without fear.

    In order to develop successfully with more than a few developers, however, you have to be able to transition code from workstation to production in such a way as to ensure that any one developers' tweaks do not break anything as they move from their customized workstation to a generic production environment.

    That would entail:

    • Good code revisioning; e.g., with CVS
    • A multi-stage release process in which
      1. Developers commit working code to CVS
      2. The (working) code changes are checked out to a collective development environment which is maintained to be like the production environment, but which developers can feel free to make changes to (e.g., to the back end)
      3. Once the code passes muster on the development server, it is migrated by someone in charge of the release process to a QA server and a walk-through is done of the release process. The QA server should be kept as close to production as possible, mirroring the back end if you've got one. After the walk-through is successfully completed, the code changes are tested on the QA server. If either the walk-through or QA process is unsuccessful, the QA server is restored to its original state and you wash-rinse-repeat until everything goes smoothly.
      4. Only after the walk-through and QA are successful on the QA server is the code pushed to the production environment.
    • A rigorous release process in which only one or two people have access to the QA and production environments. All code changes must go through that person or persons before being considered for release.

    As a side benefit, your developers become much more familiar with the environment that their code runs in. They are better off because they have improved professionally, and you are better off because now you have a whole team that can help you diagnose critical production problems rather than just a handful of people.

    Hope this helps :-)

      Having once worked at a company that followed the above approach, I wholeheartedly recommend it. Developers have a local dev environment. Code is committed to HEAD in CVS. The communal dev server is updated from CVS daily or more often, so breakage is noticed quickly. Then QA does it's thing, and updating the live site is as simple as 'cvs update -r whatever; check db schema; apachectl restart'

      I was the newest developer at the company, and I updated the production servers more than once. In fact, it was so easy I was at first a bit scared about the whole process. (The boss didn't like production to throw 500 errors... Imagine that! :-)