I haven't tested yet, but this is the general idea:

In the directory you want left alone, create an autohandler:

<%flags>
  inherit => undef
</%flags>
<% $m->call_next %>
[download]

Alternatively, you can create an .htaccess file in the desired directory (or <Directory> block in httpd.conf) that has a RemoveHandler directive.