Just remove them. If they're not empty, rmdir will fail. In this case, you don't care that it fails, ignore the error, and keep going.

use File::Find;
finddepth(sub { rmdir $_ if -d }, $some_path);
[download]

Update: added 'depth' as per merlyn's correction.