Well then, maybe next time you ask a question, you will specify more details. How was I supposed to know you had one entry per line? I just assumed comma-separated because that's the closest assumption I could make out of the original posting. In any case, the code I provided will work for a one-item-per-line file anyway. But here's a refactored version of the whole thing:
#!/usr/bin/perl -w
use strict;
chdir('dir1');
open(my $fh, '<', 'list.name') or die("open() failed: $!");
my %all = map { $_ => undef } grep { -d } glob("*");
delete($all{$_}) for ( map { chomp; $_ } <$fh> );
system('rm', '-ir', $_) for (keys(%all));
__END__
format of dir1/list.name:
dir1
dir2
dir3
dir4
dirx