Please remember that distribution names and module names need not be the same. For example, the distribution "ack" contains the module "App::Ack", as well as the program "ack".

So yeah, go ahead with the "dummy .pm file". As someone else pointed out already, App::Ack exists only to be able to have $App::Ack::VERSION which the CPAN shell can check so that people can easily do updates.