The variety of answers you've already gotten just shows how broad the TMTOWTDI principle actually is in Perl. I would do something entirely different than has yet been suggested--might not be the best, but it has worked for me in similar situations a number of times.

my $check = sprintf("%0.2d", $x);
next if ('01,02,05,06,09,10,41,42,43,44,45,46,47,48,49,50,51,52,53,54,
+55,56' =~ m/$check/);
[download]

Obviously, if the list gets too long, this might not be practical. As written, this accommodates one- or two-digit numbers (though this could be tailored for larger ones, with the addition of more leading zeros). But for a list such as you suggested, it might work well enough--better than taking time to hash it all in terms of its simplicity, if not its efficiency.