You don't use "bydepth" option. Which means that you can cancel processing of any subdirectory by setting $File::Find::prune=1. So, in your "wanted" function check if the file should be excluded from the processing and then set $File::Find::prune for the directories that should be excluded and simply ignore the files that should be excluded. Something like

sub my_wanted{
    if(exists $skip_them{$_})
    {
        $File::Find::Prune = 1 if -d $_;
        return;
    }
    do_process($_);
}
[download]