G'day Cody Fendant,

This appears to be a known bug: Bug #32905 for Perl-Tidy: Broken handling of UTF-8 strings

I didn't read the full discussion. A number of patches have been submitted. I'll leave you to investigate further.

The patches are to convert UTF-8 to an 8-bit encoding, which doesn't work -- the real problem goes much deeper since Perl::Tidy breaks down line input into bytes and processes things byte-by-byte (using maps for the "known 256 chars).... Such maps don't scale so well when you move up to the number of chars in Unicode. It looked to be close to a rewrite of the core code.

It's been an open bug since 2008 and doesn't look easy to fix so it hasn't been.

Expect to see it handle unicode sometime right after perlmonks does. ;-)

++ Thanks for the feedback.

— Ken

I can't add command-line flags like -CIO, that's not allowed.

Forget that for a moment, does it work if you add the flags?

I've tried adding binmode STDOUT, ":utf8" and binmode STDIN, ":utf8" to the BEGIN block but that hasn't changed anything.

That hints there problem isn't in perl land ... what happens if you use Data::Dumper::dd instead of print?

Forget that for a moment, does it work if you add the flags?

I should have been more specific. It's not allowed in the sense that the script will die with an error message "too late to add -C flag".

oh, right, forgot about that

What about the Dumper?

If i try

perl -CSD -le " print qq{use utf8;\nprint qq{I \x{2665} Perl\n};\n}; "
+ >foo.pl
[download]

I get

use utf8;
print qq{I ♥ Perl
};

If I run that through perltidy or your program its unchanged

cmd.exe doesn't know how to display the unicode without a chcp, but the bytes are the same

If I add to your program

    use Data::Dump qw/pp /;
    print STDERR pp("$output_string");
[download]

I get unchanged correct result as expected

"use utf8;\nprint qq{I \xE2\x99\xA5 Perl\n};\n\n"
[download]

by default cmd.exe does not unicode

$ chcp
Active code page: 437

$ type foo.tdy
use utf8;
print qq{I ΓΦΡ Perl
};

If I change it I get a heart

$ chcp 65001
Active code page: 65001

$ type foo.tdy
use utf8;
print qq{I ♥ Perl
};