Re: Losing Bits with Pack/Unpack

I am trying to pack bytes into as few unicode characters as possible. Consider the Twitter 140 character limit, which allows byte-heavy unicode characters.

I don't think it's as straightforward as this. Consider that there are lots of unassigned Unicode code points, different kinds of whitespace, nonprintables, and so on, all of which may or may not be removed/replaced/folded depending on where you're posting this text. This means you'd have to look at the character properties, select only those characters that are likely to work correctly, and make a mapping.

I believe I can squeeze 3 8-bit ascii characters into a single 20-bit unicode character

Well, to nitpick, ASCII is 7 bits, and 3*7 is 21, while Unicode code points range from 0 to 0x10FFFF (1114111), which won't fit 2^21==2097152. If you exclude ASCII control characers, for example, you'd also be excluding Tab, CR, and LF. afoken went into more details.

Comment on Re: Losing Bits with Pack/Unpack

Replies are listed 'Best First'.

Re^2: Losing Bits with Pack/Unpack
by ikegami (Patriarch) on Sep 18, 2020 at 08:54 UTC

ASCII has 30 control characters and four whitespace characters (SPACE, TAB, CR and LF). If you forgo support for control characters, TAB and CR (but keep space and LF), you end up with 0x60 characters. This isn't a power of 2 (which would help make things simple and very efficient), but it's still a nice number (3/4 of 2^7).

That would require an address space of 0x60^3 = 884,736 (0xD_8000) code points. That's a fair bit smaller than the 1,114,112 (0x11_0000) code points Unicode supports.

Of those, some are best avoided. I would avoid at least the following:

High surrogates (1024)
Lo surrogates (1024)
Non-characters (66)
U+FFFD
Control characters, which includes U+FEFF (226)

That's only 2,341 and we have a buffer of 229,376. Golden!

Mapping the 3 ASCII characters (with the limitations mentioned above) unto only "safe" characters won't be nice and easy, but it is doable.

[reply]


Your skill will accomplish what the force of many cannot
	PerlMonks