I explain how I handled this in my mod_perl Picture Handler column. If your handler gets a directory for a URI that doesn't end in slash, it must decline it unless it's prepared to handle the redirect that mod_dir normally does. In short:

use Apache::Constants qw(:common DIR_MAGIC_TYPE);
...
sub handler {
  my $r = shift;
  return DECLINED
    if $r->content_type eq DIR_MAGIC_TYPE
    and not $r->uri =~ m{/$};
  ...
}
[download]

If you don't want to handle directories at all, leave off the latter part of the check: just return DECLINED for anything that is DIR_MAGIC_TYPE.