The point is, that you have an empty skeleton directory tree. The toppermost directories are not themselves empty as they contain directories.

Here is a technique I use in one of my module tests for removing the directory 'test' and anything below:

find( {
    bydepth => 1,
    wanted => sub { if (-d $_) {
                rmdir $_;
            } else {
                1 while unlink $_; 
            }
               },
    }, 'test') if -d 'test';

rmdir 'test';
[download]

In case you are wondering about 1 while unlink, check out the node.