If you have enough memory, and you're using Apache, you may want to look into using mod_perl.

Although there may be some quirks to deal with, it'll keep one copy of each module in memory, that all of the CGIs can share, so there isn't the loading and teardown for each page call.

Of course, this won't solve the general case of the issue that you're seeing (the answer to that is to not 'use' modules up front, but 'require' them as you need them, and then import the functions that you need, (if any, as you said you were using fully qualified functions).

As you've said you're using fully qualified function calls, you may have a leg up on the second approach.

One solution that I didn't see mentioned is to replace use with require(s) at the first point(s) where that module's feature(s) are actually needed. This is likely an inappropriate solution for many cases, but it can be a simple and very effective solution in some cases.

If this is the case, I'd think that fixing a few widely used packages might reduce the problem a lot; thereby reducing the time constraints such that you'd be able to clean up most of the package collection over a more extended period of time.

-Scott

Update: For a more specific suggestion, I'd call your attention to this:

"some of the problem packages it turns out that many of the included packages are included for a function that only gets used in a few specifc places

Excellent advice.

Yep, the real answer is to re-factor the package that pulls in half the other packages - that just doesn't sound right!

I think there are three approaches outside of re-factoring:

• AutoLoader and autosplit ala POSIX.
• SelfLoader ala Text::Balanced
• roll your own loader like CGI.

Some caveats about SelfLoader. We've been using it in production here at my company for awhile and it has a few gotchas:

1) Debugging becomes more difficult as line numbers aren't reperesnted correctly, nor the actual error. Often the error will be presented in the highest-up module used in your tree. I recommend having a script that will comment out the selfloader code while you are in development and put them back in when you are about to put things in production.

2) Many editors get confused by SelfLoader because your actual subroutines sit below the __DATA__ declaration. I don't know how to trick Emacs or VI into seeing these correctly, but JEdit maintains its highlighting correctly.

The speed gains in a non mod-perl environment are nice though.

benefits:
1. If I remember correctly, we saw noticeably lower memory usage and a just-barely quicker startup time.

2. Merging changes, patching old releases, and working together with a large team is much easier. Instead of constantly merging changes to the same large files, now we're typically each working on tiny files that don't affect one another.

3. Copying code from one script to another doesn't result in annoying errors when we forget to require the right lib files. We require one lib file with the AUTOLOAD and it takes care of the rest, including nice error reporting and logging when something goes wrong (sub not found, sub doesn't compile, etc.).

I've really grown to like our current system, more because of #2 and #3 than #1.

drawbacks:
Once in a while you're going to run into wierd things caused by the autoload. I think we had a problem with sort subs in other packages, for example:

package A;
my @sorted = sort B::notYetLoaded @unsorted;
[download]

We've found workarounds for all of the problems, but each one took time.

Also, I recently tried to use inheritance (we only recently started doing much object-oriented programming). Working with our AUTOLOAD (while trying to preserve its behavior for all our existing code) was ugly enough that I gave up inheritance for now.

-Joe