unicode keys in dicts

[Jiba]

Hi all,

is the following behaviour normal :
[color=blue][color=green][color=darkred]
>>> d = {"é" : 1}
>>> d["é"][/color][/color][/color]
1[color=blue][color=green][color=darkred]
>>> d[u"é"][/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
KeyError: u'\xe9'


it seems that "é" and u"é" are not considered as the same key (in Python
2.3.3). Though they have the same hash code (returned by hash()).

And "e" and u"e" (non accentuated characters) are considered as the same
!

Jiba

[Jeff Epler]

Re: unicode keys in dicts

>>> chr(0xe9) == unichr(0xe9)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeError: ASCII decoding error: ordinal not in range(128)

unequal objects can hash to the same value. Your two keys are not
equal (in fact, you can't even compare them on my system). They would
be comparable but not equal on many systems, for instance one where the
system's encoding is Microsoft's CP850.

You can misconfigure your system to assume that byte strings are in (eg)
iso-8859-1 encoding by changing site.py.

Jeff

[Peter Hansen]

Re: unicode keys in dicts

Jiba wrote:[color=blue]
>
> is the following behaviour normal :
>[color=green][color=darkred]
> >>> d = {"é" : 1}
> >>> d["é"][/color][/color]
> 1[color=green][color=darkred]
> >>> d[u"é"][/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> KeyError: u'\xe9'
>
> it seems that "é" and u"é" are not considered as the same key (in Python
> 2.3.3). Though they have the same hash code (returned by hash()).
>
> And "e" and u"e" (non accentuated characters) are considered as the same
> ![/color]

Well, "e" and u"e" _are_ the same character, while the unicode that comes
from decoding the "é" representation is entirely dependent on which codec
you use for the decoding. It is only the same as u"é" when decoded using
certain codecs, most likely. ASCII is 7-bit only, so the "é" value is
not legal in ASCII, which is likely your default encoding.

For example, try "é".decode( 'iso-8859-1') and you will probably get the
unicode value you were expecting.

I'm not the best to answer this, but I would at least say that the above
behaviour is considered "normal", though it can be surprising to those
of us not expert in Unicode issues...

-Peter