The rmdir FILENAME command will only delete directories that are empty. If it succeeds it returns true, otherwise it returns false and sets $! (errno). Maybe you can look into using that as a flag?
Comment on Re: checking to see if a directory is empty