Just stick the directory names in a hash and test against them after doing a readdir. If they do not exist in the hash, rmdir them. The caveat is that rmdir will not delete directories that are not empty... you will need to do a bit of recursion to follow the directories into their depths and rmdir/unlink all that is within. I'll leave that part to you.

my $file = 'list.name';
my @keepers;

open FILE, $file or die $!;
{
local $/ = ',';
my @keepers = <FILE>;
}
close FILE;

my %keepers = map { $_ => 1 } @keepers;

opendir(DIR, "./") or die $!;
my %current_dirs = map { $_ => 1 } readdir(DIR);
closedir DIR;

foreach my $dir (keys %current_dirs) {
   next unless (-d $dir);
   rmdir($dir) unless $keepers{$dir};
}
[download]

Again, the implementation is up to you, but that's a generalized framework for how it can be done.