Having used Date::Manip, that would be my first instinct. But, as you've read, it doesn't benchmark particularly fast (at least partially because it's written purely in Perl).

For my needs, though, (and yours too, I would guess), unless you're running loops of date calculations, the ease of use of said module will outweigh the speed hit you may take while using it.