If you put print "$_\n"; at the top of your for loop you'll see what's going on.

readdir returns the file names without the path of the directory you opened, so you've got to open "$dir/$_" as well as skipping any directories.

I generally find glob much easier and quicker to use than readdir.