How do I display unicode-paths?

[Pettersen, Bjorn S]

I've been trying to stay blissfully unaware of Unicode, however now it seems like it's my turn. From the outside it seems like a rather massive subject, so any pointers as to where I should _start_ reading would be appreciated. The usecase is a class:

class path(object):
...
def __str__(self):
return self.pathstr.en code(???)

the question is what to put at ??? to be most useful to programmers/end users?

Background: I've got a directory with a file called 'bæ', B-AE, (which everyone knows is what Norwegian sheep say :-). If I type u'bæ' at the Python command prompt, it returns:

u'b\x91'

(WinXP, 2.3.2, regular command window). With a little trial and error I found:
[color=blue][color=green][color=darkred]
>>> print u'b\x91'.encode ('latin1')[/color][/color][/color]
bæ

although
[color=blue][color=green][color=darkred]
>>> print u'b\x91'[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "e:\python23\li b\encodings\cp4 37.py", line 18, in encode
return codecs.charmap_ encode(input,er rors,encoding_m ap)
UnicodeEncodeEr ror: 'charmap' codec can't encode character '\u91' in position 1:
character maps to <undefined>

not sure I'm understanding this, and when I call:

os.listdir(os.g etcwdu())

I get back

u'b\xe6'

which isn't making much sense either...

help?!

-- bjorn

[Martin v. Löwis]

Re: How do I display unicode-paths?

Pettersen, Bjorn S wrote:[color=blue]
> I've been trying to stay blissfully unaware of Unicode, however now it seems like it's my turn. From the outside it seems like a rather massive subject, so any pointers as to where I should _start_ reading would be appreciated. The usecase is a class:
>
> class path(object):
> ...
> def __str__(self):
> return self.pathstr.en code(???)
>
> the question is what to put at ??? to be most useful to programmers/end users?[/color]

It is not possible to put something "most useful" here. If you want to
convert a Unicode string to a byte string, you typically do so because
you need to put data into a byte-oriented channel (such as a file, a
socket, or a terminal). You *do* need to know the encoding of that
channel to convert the Unicode object. Unfortunately, __str__ has no way
of knowing where data will be put.

If you can assume that the bytes returned are displayed to the user on
the local system, you can try locale.getprefe rredencoding(). This should
work in most cases, with the notable exception of a cmd.exe window on MS
Windows: the terminal uses an encoding *different* from the user's
preferred encoding, and it is impossible to know in advance what that
encoding will be.

So in short: Just don't convert the Unicode string into a byte string,
until you know exactly where it will be displayed.
[color=blue]
> Background: I've got a directory with a file called 'bæ', B-AE, (which everyone knows is what Norwegian sheep say :-). If I type u'bæ' at the Python command prompt, it returns:
>
> u'b\x91'[/color]

Unfortunately, this did not work very well: Try the same in IDLE, and
you will get u'b\xe6'

Unicode character U+00E6 really is LATIN SMALL LETTER AE:
[color=blue][color=green][color=darkred]
>>> u"\N{LATIN SMALL LETTER AE}"[/color][/color][/color]
u'\xe6'

Unicode character U+0091 is *not* AE, it is some (useless) control
character. The problem is that the windows console was using MS CP850,
but Python was interpreting it as Latin-1 (see PEP 263).

Print u"\xe6" on the console should work fine, in Python 2.3 (since
Python finds out, at the time of putting out the bytes, what the
encoding of the console is).

Regards,
Martin