Surprisingly, the trick only works for open, not binmode.

use strict;
use warnings;

sub dump_layers(*) {
   my @layers = PerlIO::get_layers($_[0]);
   print STDERR "@layers\n";
}

my $file = 'temp';

{
   open(my $fh, '>:raw:encoding(ucs-2le):crlf:utf8', $file) or die;
   dump_layers($fh);
}

{
   open(my $fh, '<:raw:encoding(ucs-2le):crlf:utf8', $file) or die;
   dump_layers($fh);
}

unlink $file;

binmode STDOUT, ':raw:encoding(ucs-2le):crlf:utf8' or die;
dump_layers STDOUT;

binmode STDIN, ':raw:encoding(ucs-2le):crlf:utf8' or die;
dump_layers STDIN;
[download]

unix encoding(UCS-2LE) utf8 crlf utf8
unix encoding(UCS-2LE) utf8 crlf utf8
unix crlf encoding(UCS-2LE) utf8
unix crlf encoding(UCS-2LE) utf8
[download]

The only solution I've found follows, but I don't like it.

binmode $fh, ':raw:pop:encoding(ucs-2le):crlf:utf8';
#                  ^^^
[download]

The issue is that :raw disables the existing :crlf layer (but doesn't remove it) when using binmode. Then, the later :crlf reactivates the earlier :crlf layer instead of adding a new layer, messing everything up.