If you're brave, you can construct a regex which you can bang your various directories against. This can be done quite simply like so:

my $skip = join ('|', map {quotemeta} keys %$config_dirs);

This will look something like "/usr/bin|/home|/var" or what have you. Now you can just go and check against this, like dragonchild suggested, but with a slight mod:

foreach (keys %$dirs)
{
    next if /^($skip)/;
    # Do stuff
}
[download]

Presumably if the value of $skip does not change within your program, you can use the /o option to only "compile once" your regular expression.