i did some trail and error method and i found that if the dir has no sub directories and it has only files then it removes the folder otherwise it's not removing the folder. Any chance to remove the folder and it's sub folder etc., in FTP.
It works for me, it deletes directory with all its subdirectories without any problems. So now question is what ftp server you're using and what version of Net::FTP? Try to update Net::FTP.