Try e-mailing the people who have made the rpms and debs, offer to host their packaging of your program if they keep it updated.
Yeah, that's probably a very sensible idea.

One more thing: I'd definitely learn how to use PAR to package the CPAN modules for your application, it will improve your Perl kung-fu greatly.
PAR looks cool, but it does look like it would require taking my perl-fu to the next level :-) I wonder if Perl 6 and Parrot will simplify this process at all. Even if PAR works great, it strikes me as kind of a bolted-on mechanism for something that might be better supported within the language itself. IIRC, Perl 6 will eliminate the problems with having to recompile XS when you upgrade perl; that will really be nice.