Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

Re: Losing Bits with Pack/Unpack

by haukex (Bishop)
on Sep 17, 2020 at 07:19 UTC ( #11121866=note: print w/replies, xml ) Need Help??

in reply to Losing Bits with Pack/Unpack

I am trying to pack bytes into as few unicode characters as possible. Consider the Twitter 140 character limit, which allows byte-heavy unicode characters.

I don't think it's as straightforward as this. Consider that there are lots of unassigned Unicode code points, different kinds of whitespace, nonprintables, and so on, all of which may or may not be removed/replaced/folded depending on where you're posting this text. This means you'd have to look at the character properties, select only those characters that are likely to work correctly, and make a mapping.

I believe I can squeeze 3 8-bit ascii characters into a single 20-bit unicode character

Well, to nitpick, ASCII is 7 bits, and 3*7 is 21, while Unicode code points range from 0 to 0x10FFFF (1114111), which won't fit 2^21==2097152. If you exclude ASCII control characers, for example, you'd also be excluding Tab, CR, and LF. afoken went into more details.

Replies are listed 'Best First'.
Re^2: Losing Bits with Pack/Unpack
by ikegami (Patriarch) on Sep 18, 2020 at 08:54 UTC

    ASCII has 30 control characters and four whitespace characters (SPACE, TAB, CR and LF). If you forgo support for control characters, TAB and CR (but keep space and LF), you end up with 0x60 characters. This isn't a power of 2 (which would help make things simple and very efficient), but it's still a nice number (3/4 of 2^7).

    That would require an address space of 0x60^3 = 884,736 (0xD_8000) code points. That's a fair bit smaller than the 1,114,112 (0x11_0000) code points Unicode supports.

    Of those, some are best avoided. I would avoid at least the following:

    • High surrogates (1024)
    • Lo surrogates (1024)
    • Non-characters (66)
    • U+FFFD
    • Control characters, which includes U+FEFF (226)

    That's only 2,341 and we have a buffer of 229,376. Golden!

    Mapping the 3 ASCII characters (with the limitations mentioned above) unto only "safe" characters won't be nice and easy, but it is doable.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11121866]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (7)
As of 2022-05-17 17:18 GMT
Find Nodes?
    Voting Booth?
    Do you prefer to work remotely?

    Results (68 votes). Check out past polls.