Smoothhound, you potentially have a big task on your hands.

Version Control

If you are not using a version control system, start using one. Check in the current version of all modules, and use this for progress. Even if you are working solo, it's still worth doing.

use strict and warnings

If this were me, I would start with a review of code quality in general. Do the modules use strict and warnings? If the answer is no, this would provide a starting point. Aim to get all the modules passing strict and not emitting warnings. See Thoughts on revisiting old code for my musings on this subject.

Review module interfaces

This may be a question of looking at documentation rather than code. In the worst case, there may be no documentation, not even comments in the code; if this is the case, write a POD section for each module describing its interface.

Module deployment

Just because a module is not going onto CPAN is no reason not to follow the standard guidelines for delivering a module. Most of the modules you are looking at were probably not generated with h2xs and friends (ExtUtils::ModuleMaker, Module::Starter). Make kits for each module or collection of modules.

Module tests

The CPAN style install mechanism has a provision for including unit tests. Write a suite of tests for each module kit.

1. Refactor.
2. Rewrite.

First thing you need, is a full understanding of the system. Feature freeze and map out everything. Every module, config file and any other dependency. Next, have a good understanding of what an ideal version of the system would be. This includes where things should go and what the system will look like.

Here's the easy part. You make the old system look like the new system. You start to do 1 of many things.

1. Run both systems in parallel, the old and the new as it is being written. This means data gets written to two places, and then parts of the software become deprecated as new ones are put in. This is not so much cut by product, but function.
2. Build the new system and old system concurently, providing hte new system to a very small controlled set of people. You can address problems, and things you won't need as the new system grows. Think of it like beta testing. This cuts across some percentage of your customers.
3. Go in a cave, rewrite the system, forget the old, then cut over. This takes a long time and is very hard to estimate anything about. Prone to a lot of people being pissed at the end due to things broken for everyone. At the same time, a lot more poeple could be happy with the new system.
4. Stepwise rebuild. It sounds like the first part, but kinda is, yet isn't. It is akin to rewriting newest nodes while RAT stays in place. One shouldn't affect the other as they are being rewritten serially. Perlmonks is more commingled in its feature division that other systems, which are larger and have more parts, such as a bank's financial system.

Uh, step 3, profit. That's about it. Other than the above, prof'ing and patching, there's not too much else you can do.

Uh, what sporty said. Sometimes you just have to step back ten yards and punt. Otherwise you wind up preserving junk and cruft that keeps slowing your development and operations as time goes on. The real problem is convincing management of this, many of whom either don't get it or don't care since they won't be in their positions that long. We're getting close to that point here at work, and we're doing a few 'stealth', fill-in projects to start the ball rolling while we can. It's not an ideal solution, but it beats ignoring the problem and hoping it will go away.

--
tbone1, YAPS (Yet Another Perl Schlub)
And remember, if he succeeds, so what.
- Chick McGee

Management is always a tricky issue. jeffa, who was told by maverick eduardo, told me, which is true: people who like you are more likely to listen. It's a psych trick that involves disonence and grouping I think. Anyway, gotta have management like you and listen to a reasonable argument.

After all, systems are like cars. If you do the upkeep very well, replace parts as they break down, use better parts when applicable, any system can become standard for the company. Otherwise, people start using the wrong oil's, the cushins look like crap, rust starts to occur... bleh.

----
Give me strength for today.. I will not talk it away..
Just for a moment.. It will burn through the clouds.. and shine down on me.

I have read and digested the recommendations in perlmodstyle but apart from the obvious things like splitting up the larger modules and making the purpose of each as tightly defined as possible is there any best practice I should be following before I embark on this? Is there anyone else out there who has had to deal with a similar problem and can share their experience with me?

I have no idea if it's the best way to do things, but when I've been faced with similar problems, I've tried to change out the instances of 'use' to 'require'. The problem lies in that you need to figure out exactly when each module gets used (and by 'used', I don't mean the 'use' command, I mean when it is actually needed by the calling program). You can also run into problems if you need for there to be functions imported -- you can either change out functions to use the full namespace, or after the require, call import() in the namespace.

Some modules are smart, and do this sort of work for you (not loading the bulk of their program until the fuctions are actually needed), so for Carp, and other modules that don't expect you to actually need them, just because they were mentioned, you don't have to worry about this trouble.

As an example -- someone may have some cases that result in debugging output being sent to the error log under certain conditions:

use Data::Dumper;
 # much later in the program
if ( $some_condition) {
  warn "DEBUG: ", Dumper ( $whatever );
}
[download]

You only need to load Data::Dumper when the need to generate the message has occurred:

if ( $some_condition ) {
  require Data::Dumper;
  warn "DEBUG: ", Data::Dumper::Dumper( $whatever );
}
[download]

As for how to clean up things ... you might want to take a look at the book 'Perl Medic' for a start -- make an archive of the working copy, then get everything happy under use warnings and use strict. Then, if you want to do what I suggested, comment out every 'use' line, and see what warnings/errors you start generating, and on which line, and add your 'require's near those lines.

Some dependancies may be so important to your module that it's better just to leave it as is. (all usage patterns are different, though, so the benefits are going to be different depending on how often you call each function that needs to check if its required dependencies have been loaded)

While the advice above on refactoring is certainly excellent, and probably the best in the long run, you may also consider using an infrustructure like Apache's mod_perl. Sometimes, the porting of a system to mod_perl can be faster than refactoring all those modules.

The advantage you gain is that mod_perl applications compile once and remain running, so your compile time imports will only hurt you once. Of course, depending on how the application is constructed, mod_perl porting might actually be more work than refactoring your modules. Still, it's something to consider.

Regarding the refactor of the modules, I must say I really do feel sorry for you, as this is a ton of work. As a first step, follow the advice above on converting function calls to use the full namespace. Do that first, and the next steps are easier. Maybe this isn't the best approach, but it's what I did the last time I ran into this issue:

1. Convert all functions to full namespace, and all 'use' to 'require' (except for core Perl and CPAN modules, which you won't need anyhow).
2. Use your favorite diagramming system (I was forced into UML, which worked fine) to analyze each module and figure out its internal and external dependencies.
3. Get the system as is into version control.
4. Create a stub module for each subroutine in each module; for example, My::BigModule::function1() will become My::Module::Function1::function1() (note the change in case). It's easier to recombine modules than to split them further, so you might as well split as far as you can up-front.
5. One subroutine at a time, copy to the new module space and update your code to reflect the new namespace (search and replace works just fine for this). Test after you move each subroutine and make note of any that aren't used by anything. Also, be sure to update a copy of your dependency diagram.
6. Figure out which subroutines are always (or frequently) loaded together, and combine them.
7. Figure out which subroutines fit conceptually together and move them in your namespace as you see fit.

Again, this may not be the best approach. It worked reasonably well for me, though; and it scales to teams, to some extent.

While the advice above on refactoring is certainly excellent, and probably the best in the long run, you may also consider using an infrustructure like Apache's mod_perl. Sometimes, the porting of a system to mod_perl can be faster than refactoring all those modules.

It also might be the right thing to do. The fact that most modules need to be loaded most of the time might not be a sign of bad design, and moving to a persistant architecture could well be the best, as well as possibly (depending on how much the current system relies on having a clean slate between runs) the simplest, solution.

If you have done some profiling and have discovered that load time is really the problem in your case, then there is a simple change that you may be able to employ to speed up load time. Have a look at autouse and/or Class::AutoUse. You can use these to prevent a module from loading until it is actually used in the program.

But this will only help with scripts that run once and exit (ie not mod_perl or PersistentPerl), and will not do anything if these scripts always use some small part of most of the modules (Since they will all need to be loaded anyway). Also, it will obviously only work if load time is really your bottleneck...

Another option would be to look at using PersistentPerl, which will load your script once, and then keep it in memory. This removes the startup cost of the scripts, but does nothing to improve execution time. Also with PersistentPerl you will need to be careful with things like global variables and memory leaks, since the process will have a long life.

- Cees

I would just second the comment that you should make sure you have done some profiling to really prove where you are spending most of your time. It's easy to point your finger at the wrong thing when optimizing, and in many cases the more you know, the more likely you are to focus on the wrong thing.