Using File::Slurp::read_dir() as in:

use File::Slurp;

if ( read_dir('/path/to/dir') == 0 ) {
   print "is empty!\n";
}
[download]

Update: using File::Slurp or Path::Class will have bad performance on a directory with a large number of entries. However, File::Find::Rule::DirectoryEmpty is addressing the problem the right way, as mentioned by Fletch, by looking for any other entry aside from "." and ".." (which means 3 readdir calls at most whatever the number of entries).