Thanks so much for all the great explanations! That really cleared things up. I also wrote a hack to solve the problem I alluded to at the end of my first post. It is extremely inefficient, but seems to properly carry out all of the conversions I'm interested in. Please rip it to shreds for me:
#!/usr/bin/perl -w
use strict;
use warnings;
#Convert annoying, worthless 'fullwidth' Latin-1 characters
#to their semi-sane normal ASCII counterparts
my(%codes,$wide,$ascii,$x);
#this is the land where the fullwidth latin-1 characters reside
for($x=65281;$x<65374;$x++){
($wide,$ascii) = make_codes($x);
$codes{$wide} = $ascii;
}
while(<>){
chomp;
foreach my $utf(keys %codes){
s/$utf/$codes{$utf}/g;
}
print $_."\n";
}
sub make_codes{
my $ud = $_[0];
my $from = ud_to_utf8hex($ud);
#subtract 65248 to get the ASCII value
my $to = ud_to_utf8hex($ud-65248);
return($from, $to);
}
sub ud_to_utf8hex{
my $ud = $_[0];
my ($b1,$b2,$b3);
if($ud >= 0 && $ud <= 127){ #basic ASCII values don't need to be a
+ltered
return(sprintf("%c",$ud));
}elsif($ud >= 2048 && $ud <= 65535){ #valid for 2048 <= $ud <= 655
+35
$b1 = 224 + sprintf("%d", ($ud/4096));
$b2 = 128 + (($ud/64) % 64);
$b3 = 128 + ($ud % 64);
}
return(sprintf("\\x\{%X\}\\x\{%X\}\\x\{%X\}",$b1,$b2,$b3));
}
I chose to use the decimal numbers for some practice with the unicode standard, and because they are easier on my brain than the hex codes.
