When you specify 'X:*' i am assuming this is not a regular expression. With that caveat, perhaps this is cleaner (or even clearer) for you?

use strict;
use warnings;

my %hash1 = ( 'X:Y' => 10, 'X:*' => 1, '*' => 1 );
my %hash2 = ( 'X:Z' => 1, 'X:*' => 20, '*' => 1 );
my %hash3 = ( 'A:B' => 1, 'A:*' => 1, '*' => 30 );

print find_level( 'X:Y', \%hash1 ), $/;
print find_level( 'X:Y', \%hash2 ), $/;
print find_level( 'X:Y', \%hash3 ), $/;


sub find_level {
    my ($item, $hash) = @_;

    for ( $item, join( ':', (split ':', $item)[0], '*' ), '*' ) {
        return $hash->{$_} if exists $hash->{$_};
    }

    return 0;
}
[download]

L-LL-L--L-LL-L--L-LL-L--
-R--R-RR-R--R-RR-R--R-RR
B--B--B--B--B--B--B--B--
H---H---H---H---H---H---
(the triplet paradiddle with high-hat)