Are the directories empty?

$ perldoc -f rmdir
       rmdir FILENAME
       rmdir   Deletes the directory specified by FILENAME if
               that directory is empty.  If it succeeds it
               returns true, otherwise it returns false and sets
               "$!" (errno).  If FILENAME is omitted, uses "$_".
[download]

Maybe you need to go through the directories and delete any files in them.

s**lil*;  $*=join'',sort split q**;  s;.*;grr; &&s+(.(.)).+$2$1+; $; =
qq-$_-;s,.*,ahc,;$,.=chop for split q,,,reverse;print for($,,$;,$*,$/)
[download]