... I do not want right appended to $class if Smart is in $text.

Your code seems to me to do what you want.

c:\@Work\Perl\monks>perl -wMstrict -le
"my @big_images = qw(xsMaRty Dull Boring);
 ;;
 my $text = 'Smart';
 ;;
 my $class  = 'svg_group';
    $class .= ' right' unless grep(/$text/i, @big_images);
 ;;
 print qq{'$class'};
 ;;
 $text = 'Xxx';
 ;;
 $class  = 'svg_group_2';
 $class .= ' left' unless grep(/$text/i, @big_images);
 ;;
 print qq{'$class'};
"
'svg_group'
'svg_group_2 left'
[download]

Update: You may also be interested in any and none in List::Util.