If performance is relevant, maybe you shouldn't be using Date::Manip.  In SHOULD I USE DATE::MANIP, it's explicitly stated that the module isn't one of the fastest.

You could try Date::Calc(::XS), for example, which is more focused on speed  (just one suggestion, there are quite a number of other date modules on CPAN...)