readdir gives not only the file names but also the directory names present inside your $handle.

open (IN, "$handle/$file") on a directory will give you an error of course ;-)

you will have to test $file for being a file before opening it.