That's not how my copy of perl is behaving:

$ ls -ld /tmp/test
drwx--x--x    2 root     root         1024 May  4 15:55 /tmp/test
$ perl -e 'opendir(D,"/tmp/test") or die "opendir failed: $!\n";'
opendir failed: Permission denied
[download]