When in doubt , decompose.

Have you tried dumping the data structure storing the result of parsing the umc file? Does it look right? If you are storing conversion rules in a hash, what happens if you look up '{i}' (or 'i') independently in the hash? Do you get the right unicode?

What 'key' do you need to lookup the unicode for '{i}'? If the data structures are storing the right information, what about the code that is supposed to be converting '{i}' into a form suitable for look-up in your encoding data structures. How is '{i}' being extracted? What happens if you separate out the code that extracts '{i}' and run it separately? Do you get a key suitable for look-up? Or do you get '{i' or 'i}' or even '{i}' when in fact you need '{i}' (or vice versa)?

I'd also investigate further why it was necessary to comment out the mapping for the left curly brace. Are right curlies also a problem? If not, what is special about left curlies? Is this a hint about how the encoding/decoding algorithm works? Perhaps there is something deeper that connects the problems with left curlies and '{i}'?

Best, beth

Howdy!

Daud_t.c contains a whole mess of declarations of constants. Each constant name includes the to and from encodings and the subsequence of bytes matched. I was able to see the sequence for {i} in there, and it appeared to be correct. The conversion process seems to be to chase pointers through a tangled mess ultimately leading to an output character. Or something along those lines. I'm doing a poor job of describing it, but I did not see anything out of line there.

I empirically determined that I needed to comment out LEFT CURLY BRACKET. My surmise is that if it was legal in its own right, it would get consumed early spoiling the matches...waitaminnit...(thinking as I type) I'll try moving that to the end to see if that allows it to catch otherwise unmatched {.

In the Daud encoding, curlies are the brackets around non-ASCII characters. They are, thus, special in this context. Hmmm...

Thanks for the thoughts.

yours,
Michael

I tried to replicate your situation, just doing the accented "A" characters from latin1 along with adding the dotless i, and I got the same results you got -- trying to decode from "{i}" to "\x{0131}" gave me an empty string, while everything else worked as expected.

I noticed that if the test string for decoding was "{i} " (note the space after the close-curly), it worked just fine (and didn't lose the space, either; any other character in that position would work as well).

Then I added one other code point using one character between curlies to see if that would behave the same way -- inverted q-mark / "{?}" -- and when this was in the ucm file, both the q-mark and the dotless i worked fine without further ado (no extra character needed in the test string).

So, I can't explain it (maybe some other monk can), but see if that works for you:

<code_set_name> "daud"
<mb_cur_min> 1
<mb_cur_max> 4
<subchar> \x3F
CHARMAP
<U0000> \x00    |0    #    NULL
...
#<U007B> \x7B   |0    #    LEFT CURLY BRACKET
<U007C> \x7C    |0    #    VERTICAL LINE
#<U007D> \x7D   |0    #    RIGHT CURLY BRACKET
...
# I included the next line, defining "{?}":
<U00BF> \x7b\x3f\x7d      |0  #  INVERTED QUESTION MARK
<U00C0> \x7b\x41\x60\x7d  |0  #  LATIN CAPITAL LETTER A WITH GRAVE
...
<U0131> \x7b\x69\x7d   |0  #  LATIN SMALL LETTER DOTLESS I
END CHARMAP
[download]

UPDATE: Regarding the issue of commenting out the curlies in the ucm file (U007b, U007d), this actually seems to me like a Good Idea™ in its own right. If some poor typist, trying to keyboard text using Daud encoding, happens to put curlies around a character or digraph that is not defined in your ucm file, a decode from that into unicode will yield "\x{fffd}..\x{fffd}" because those particular curlies cannot be decoded (and whatever was between them will be left unchanged). It's just good to know for sure how to identify errors of this kind.

Howdy!

Thanks for those findings! There is one other potential case (at present) where there could be a single character between {}, but at present, {i} is the only one I'm using.

I've long had a module that handled this conversion, but it was a standalone thing, not integrated into the Encode system. A friend wanted to be able to systematically recode file and directory names between UTF-8 and Daud. He found 'convmv' which turns out to be a Perl program that uses Encode to do the heavy lifting. That motivated me to once again try to make Daud work in the Encode framework, leading here.

yours,
Michael

I tried a bunch of other variations on the same sort of theme. I have only taken a quick glance at the C code created by enc2xs (and the differences in C code resulting from different variations of ucm), and I still don't understand exactly what's wrong with this process. But based on what I've seen, the symptom appears to be something like this:

For any mapping that includes strings of 3 or more "non-unicode" bytes equating to unicode code points, the arrangement of definitions in the ucm file should have at least one of the 3-byte strings before any of the longer string definitions. That is:

...
<Uhhhh>  \xhh\xhh\xhh  |0   # first mention of any multi-byte mapping
...
<Uhhhh>  \xhh\xhh\xhh\xhh |0
...
[download]

The above arrangement will work, whereas the arrangement below will have the problem as described in the OP:

...
<Uhhhh>  \xhh\xhh\xhh\xhh |0  # first mention of any multi-byte mappin
+g
...
<Uhhhh>  \xhh\xhh\xhh  |0
...
[download]

(updated to remove spurious spaces)

Bear in mind that the ucm definitions don't need to be in unicode code-point order -- enc2xs doesn't care about the ordering (except when it comes to triggering this one strange little bug).

It probably applies to any mapping involving 2-byte strings as well, (that would seem logical), but I haven't tested that. In a nutshell, try ordering your definitions with respect to how long the encoded byte strings are, or at least put one instance of a 3-byte string mapping ahead of all instances of 4-byte mappings.

(So it was just "coincidental" that I chose q-mark for the initial attempt in my first reply -- it only worked because it just happened to be conventionally placed above the accented letters.)