Collect the references in an array and create your own hash from them:

my @configs;
push @configs, Config::Auto::parse( $_ ) foreach @files;
my $SETS = { map {%$_} @configs };
[download]

You could combine the above into a oneliner yet that might obfuscate the code a bit.