There are, of course, going to be boundary cases that don't work as expected as soon as you start playing with allowing undoubled double-quotes inside of a format that expects them doubled. However Text::xSV allows you to define arbitrary filters that it preprocesses text with, and should do a reasonable job on the above with the following filter:
sub {
my $line = shift;
$line =~ s/\r$//;
$line =~ s/"(.)/""$1/g;
$line =~ s/"?,"?/,/g;
return $line;
}
Yes, there is some fragility, but it should be at least moderately hard to trigger.