Is this is telling me to use the Factory pattern or at least implement plugins?

not at all!
the number of modules you need for an application depends mainly on the application's intentions and purposes. however, you will feel the need for optimizing class designs - but that's another question...
just to calm down your concerns (and not for bragging!): in the application that i'm currently working on, the apache-kyrie named 'appstart.pl' uses about 45 classes, two of them allocating around 250kB of static memory.