Well it does seem to populate $! in at least some cases. I have an empty directory called "foo":

$ perl -e 'opendir DIR, "foo" or die "$!"; @c=readdir DIR; print "@c\n
+"; closedir DIR; @c=readdir DIR or die "$!"'
. ..
Bad file descriptor at -e line 1.
[download]

I also got the following $! error in my quick tests under the debugger:

  DB<8> @c = readdir DIR or die "$!"
Illegal seek at (eval 12)[/usr/lib/perl5/5.10/perl5db.pl:638] line 2.
[download]