"is not working"

Is not the (verbatim) error message or warning you got, not is it a description that helps us diagnose your problem. Please emend your node.

But, on a WAG, are the directories-to-delete empty?

perldoc -f rmdir tells you:

Deletes the directory specified by FILENAME if that directory is empty. If it succeeds it returns true, otherwise it returns false and sets $! (errno). If FILENAME is omitted, uses $_.
To remove a directory tree recursively ("rm -rf" on Unix) look at the "rmtree" function of the File::Path module.