WTF? Printing unicode strings

[Ron Garret]

>>> u'\xbd'
u'\xbd'[color=blue][color=green][color=darkred]
>>> print _[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

[John Salerno]

Re: WTF? Printing unicode strings

Ron Garret wrote:[color=blue][color=green][color=darkred]
>>>> u'\xbd'[/color][/color]
> u'\xbd'[color=green][color=darkred]
>>>> print _[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
> position 0: ordinal not in range(128)[/color]

Not sure if this really helps you, but:
[color=blue][color=green][color=darkred]
>>> u'\xbd'[/color][/color][/color]
u'\xbd'[color=blue][color=green][color=darkred]
>>> print _[/color][/color][/color]
½[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

[Fredrik Lundh]

Re: WTF? Printing unicode strings

Ron Garret wrote:
[color=blue][color=green][color=darkred]
>>>> u'\xbd'[/color][/color]
> u'\xbd'[color=green][color=darkred]
>>>> print _[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
> position 0: ordinal not in range(128)[/color]

so stdout on your machine is ascii, and you don't understand why you
cannot print a non-ascii unicode character to it? wtf?

</F>

[Ron Garret]

Re: WTF? Printing unicode strings

In article <mailman.5904.1 147987926.27775 .python-list@python.org >,
Fredrik Lundh <fredrik@python ware.com> wrote:
[color=blue]
> Ron Garret wrote:
>[color=green][color=darkred]
> >>>> u'\xbd'[/color]
> > u'\xbd'[color=darkred]
> >>>> print _[/color]
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in ?
> > UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
> > position 0: ordinal not in range(128)[/color]
>
> so stdout on your machine is ascii, and you don't understand why you
> cannot print a non-ascii unicode character to it? wtf?
>
> </F>[/color]

I forgot to mention:
[color=blue][color=green][color=darkred]
>>> sys.getdefaulte ncoding()[/color][/color][/color]
'utf-8'[color=blue][color=green][color=darkred]
>>> print u'\xbd'[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

[Robert Kern]

Re: WTF? Printing unicode strings

Ron Garret wrote:
[color=blue]
> I forgot to mention:
>[color=green][color=darkred]
>>>>sys.getdefa ultencoding()[/color][/color]
>
> 'utf-8'[/color]

A) You shouldn't be able to do that.
B) Don't do that.
C) It's not relevant to the encoding of stdout which determines how unicode
strings get converted to bytes when printing them:
[color=blue][color=green][color=darkred]
>>> import sys
>>> sys.stdout.enco ding[/color][/color][/color]
'UTF-8'[color=blue][color=green][color=darkred]
>>> sys.getdefaulte ncoding()[/color][/color][/color]
'ascii'[color=blue][color=green][color=darkred]
>>> print u'\xbd'[/color][/color][/color]
½

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Ron Garret]

Re: WTF? Printing unicode strings

In article <mailman.5906.1 147989402.27775 .python-list@python.org >,
Robert Kern <robert.kern@gm ail.com> wrote:
[color=blue]
> Ron Garret wrote:
>[color=green]
> > I forgot to mention:
> >[color=darkred]
> >>>>sys.getdefa ultencoding()[/color]
> >
> > 'utf-8'[/color]
>
> A) You shouldn't be able to do that.[/color]

What can I say? I can.
[color=blue]
> B) Don't do that.[/color]

OK. What should I do instead?
[color=blue]
> C) It's not relevant to the encoding of stdout which determines how unicode
> strings get converted to bytes when printing them:
>[color=green][color=darkred]
> >>> import sys
> >>> sys.stdout.enco ding[/color][/color]
> 'UTF-8'[color=green][color=darkred]
> >>> sys.getdefaulte ncoding()[/color][/color]
> 'ascii'[color=green][color=darkred]
> >>> print u'\xbd'[/color][/color]
> 1⁄2[/color]

OK, so how am I supposed to change the encoding of sys.stdout? It comes
up as US-ASCII on my system. Simply setting it doesn't work:
[color=blue][color=green][color=darkred]
>>> import sys
>>> sys.stdout.enco ding='utf-8'[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: readonly attribute[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

rg

[Robert Kern]

Re: WTF? Printing unicode strings

Ron Garret wrote:[color=blue]
> In article <mailman.5906.1 147989402.27775 .python-list@python.org >,
> Robert Kern <robert.kern@gm ail.com> wrote:
>[color=green]
>>Ron Garret wrote:
>>[color=darkred]
>>>I forgot to mention:
>>>
>>>
>>>>>>sys.getde faultencoding()
>>>
>>>'utf-8'[/color]
>>
>>A) You shouldn't be able to do that.[/color]
>
> What can I say? I can.[/color]

See B).
[color=blue][color=green]
>>B) Don't do that.[/color]
>
> OK. What should I do instead?[/color]

See below.
[color=blue][color=green]
>>C) It's not relevant to the encoding of stdout which determines how unicode
>>strings get converted to bytes when printing them:
>>[color=darkred]
>>>>>import sys
>>>>>sys.stdout .encoding[/color]
>>
>>'UTF-8'
>>[color=darkred]
>>>>>sys.getdef aultencoding()[/color]
>>
>>'ascii'
>>[color=darkred]
>>>>>print u'\xbd'[/color]
>>
>>1⁄2[/color]
>
> OK, so how am I supposed to change the encoding of sys.stdout? It comes
> up as US-ASCII on my system. Simply setting it doesn't work:[/color]

You will have to use a terminal that accepts UTF-8.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Serge Orlov]

Re: WTF? Printing unicode strings

Ron Garret wrote:[color=blue]
> In article <mailman.5906.1 147989402.27775 .python-list@python.org >,
> Robert Kern <robert.kern@gm ail.com> wrote:
>[color=green]
> > Ron Garret wrote:
> >[color=darkred]
> > > I forgot to mention:
> > >
> > >>>>sys.getdefa ultencoding()
> > >
> > > 'utf-8'[/color]
> >
> > A) You shouldn't be able to do that.[/color]
>
> What can I say? I can.
>[color=green]
> > B) Don't do that.[/color]
>
> OK. What should I do instead?[/color]

Exact answer depends on what OS and terminal you are using and what
your program is supposed to do, are you going to distribute the program
or it's just for internal use.

[Ron Garret]

Re: WTF? Printing unicode strings

In article <1147992722.970 761.220840@j73g 2000cwa.googleg roups.com>,
"Serge Orlov" <Serge.Orlov@gm ail.com> wrote:
[color=blue]
> Ron Garret wrote:[color=green]
> > In article <mailman.5906.1 147989402.27775 .python-list@python.org >,
> > Robert Kern <robert.kern@gm ail.com> wrote:
> >[color=darkred]
> > > Ron Garret wrote:
> > >
> > > > I forgot to mention:
> > > >
> > > >>>>sys.getdefa ultencoding()
> > > >
> > > > 'utf-8'
> > >
> > > A) You shouldn't be able to do that.[/color]
> >
> > What can I say? I can.
> >[color=darkred]
> > > B) Don't do that.[/color]
> >
> > OK. What should I do instead?[/color]
>
> Exact answer depends on what OS and terminal you are using and what
> your program is supposed to do, are you going to distribute the program
> or it's just for internal use.[/color]

I'm using an OS X terminal to ssh to a Linux machine.

But what about this:
[color=blue][color=green][color=darkred]
>>> f2=open('foo',' w')
>>> f2.write(u'\xFF ')[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xff' in
position 0: ordinal not in range(128)[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

That should have nothing to do with my terminal, right?

I just found http://www.amk.ca/python/howto/unicode, which seems to be
enlightening. The answer seems to be something like:

import codecs
f = codecs.open('fo o','w','utf-8')

but that seems pretty awkward.

rg

[Robert Kern]

Re: WTF? Printing unicode strings

Ron Garret wrote:
[color=blue]
> I'm using an OS X terminal to ssh to a Linux machine.[/color]

Click on the "Terminal" menu, then "Window Settings...". Choose "Display" from
the combobox. At the bottom you will see a combobox title "Character Set
Encoding". Choose "Unicode (UTF-8)".
[color=blue]
> But what about this:
>[color=green][color=darkred]
>>>>f2=open('fo o','w')
>>>>f2.write(u' \xFF')[/color][/color]
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xff' in
> position 0: ordinal not in range(128)
>
> That should have nothing to do with my terminal, right?[/color]

Correct, that is a different problem. f.write() expects a string of bytes, not a
unicode string. In order to convert unicode strings to byte strings without an
explicit .encode() method call, Python uses the default encoding which is
'ascii'. It's not easily changeable for a good reason. Your modules won't work
on anyone else's machine if you hack that setting.
[color=blue]
> I just found http://www.amk.ca/python/howto/unicode, which seems to be
> enlightening. The answer seems to be something like:
>
> import codecs
> f = codecs.open('fo o','w','utf-8')
>
> but that seems pretty awkward.[/color]

<shrug> About as clean as it gets when dealing with text encodings.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Paul Boddie]

Re: WTF? Printing unicode strings

Ron Garret wrote:[color=blue]
>
> But what about this:
>[color=green][color=darkred]
> >>> f2=open('foo',' w')
> >>> f2.write(u'\xFF ')[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xff' in
> position 0: ordinal not in range(128)[color=green][color=darkred]
> >>>[/color][/color]
>
> That should have nothing to do with my terminal, right?[/color]

Correct. But first try to answer this: given that you want to write the
Unicode character value 255 to a file, how is that character to be
represented in the file?

For example, one might think that one could just get a byte whose value
is 255 and write that to a file, but what happens if one chooses a
Unicode character whose value is greater than 255? One could use two
bytes or three bytes or as many as one needs, but what if the lowest 8
bits of that value are all set? How would one know, if one reads a file
back and gets a byte whose value is 255 whether it represents a
character all by itself or is part of another character's
representation? It gets complicated!

The solution is that you choose an encoding which allows you to store
the characters in the file, thus answering indirectly the question
above: encodings determine how the characters are represented in the
file and allow you to read the file and get back the characters you put
into it. One of the most common encodings suitable for the storage of
Unicode character values is UTF-8, which has been designed with the
above complications in mind, but as long as you remember to choose an
encoding, you don't have to think about it: Python takes care of the
difficult stuff on your behalf. In the above code you haven't made that
choice.

So, to answer the above question, you can either...

* Use the encode method on Unicode objects to turn them into plain
strings, then write them to a file - at that point, you are
writing specific byte values.
* Use the codecs.open function and other codecs module features to
write Unicode objects directly to files and streams - here, the
module's infrastructure deals with byte-level issues.
* If you're using something like an XML library, you can often pass a
normal file or stream object to some function or method whilst
stating the output encoding.

There is no universally correct answer to which encoding should be used
when writing Unicode character values to files, contrary to some
beliefs and opinions which, for example, lead to people pretending that
everything is in UTF-8 in order to appease legacy applications with the
minimum of tweaks necessary to stop them from breaking completely.
Thus, Python doesn't make a decision for you here.

Paul

[Ron Garret]

Re: WTF? Printing unicode strings

In article <mailman.5917.1 147995329.27775 .python-list@python.org >,
Robert Kern <robert.kern@gm ail.com> wrote:
[color=blue]
> Ron Garret wrote:
>[color=green]
> > I'm using an OS X terminal to ssh to a Linux machine.[/color]
>
> Click on the "Terminal" menu, then "Window Settings...". Choose "Display"
> from
> the combobox. At the bottom you will see a combobox title "Character Set
> Encoding". Choose "Unicode (UTF-8)".[/color]

It was already set to UTF-8.
[color=blue][color=green]
> > But what about this:
> >[color=darkred]
> >>>>f2=open('fo o','w')
> >>>>f2.write(u' \xFF')[/color]
> >
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in ?
> > UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xff' in
> > position 0: ordinal not in range(128)
> >
> > That should have nothing to do with my terminal, right?[/color]
>
> Correct, that is a different problem. f.write() expects a string of bytes,
> not a
> unicode string. In order to convert unicode strings to byte strings without
> an
> explicit .encode() method call, Python uses the default encoding which is
> 'ascii'. It's not easily changeable for a good reason. Your modules won't
> work
> on anyone else's machine if you hack that setting.[/color]

OK.
[color=blue][color=green]
> > I just found http://www.amk.ca/python/howto/unicode, which seems to be
> > enlightening. The answer seems to be something like:
> >
> > import codecs
> > f = codecs.open('fo o','w','utf-8')
> >
> > but that seems pretty awkward.[/color]
>
> <shrug> About as clean as it gets when dealing with text encodings.[/color]

OK. Thanks.

rg

[Serge Orlov]

Re: WTF? Printing unicode strings

Ron Garret wrote:[color=blue]
> In article <1147992722.970 761.220840@j73g 2000cwa.googleg roups.com>,
> "Serge Orlov" <Serge.Orlov@gm ail.com> wrote:
>[color=green]
> > Ron Garret wrote:[color=darkred]
> > > In article <mailman.5906.1 147989402.27775 .python-list@python.org >,
> > > Robert Kern <robert.kern@gm ail.com> wrote:
> > >
> > > > Ron Garret wrote:
> > > >
> > > > > I forgot to mention:
> > > > >
> > > > >>>>sys.getdefa ultencoding()
> > > > >
> > > > > 'utf-8'
> > > >
> > > > A) You shouldn't be able to do that.
> > >
> > > What can I say? I can.
> > >
> > > > B) Don't do that.
> > >
> > > OK. What should I do instead?[/color]
> >
> > Exact answer depends on what OS and terminal you are using and what
> > your program is supposed to do, are you going to distribute the program
> > or it's just for internal use.[/color]
>
> I'm using an OS X terminal to ssh to a Linux machine.[/color]

In theory it should work out of the box. OS X terminal should set
enviromental variable LANG=en_US.utf-8, then ssh should transfer this
variable to Linux and python will know that your terminal is utf-8.
Unfortunately AFAIK OS X terminal doesn't set that variable and most
(all?) ssh clients don't transfer it between machines. As a workaround
you can set that variable on linux yourself . This should work in the
command line right away:

LANG=en_US.utf-8 python -c "print unichr(0xbd)"

Or put the following line in ~/.bashrc and logout/login

export LANG=en_US.utf-8

[Robert Kern]

Re: WTF? Printing unicode strings

Ron Garret wrote:[color=blue]
> In article <mailman.5917.1 147995329.27775 .python-list@python.org >,
> Robert Kern <robert.kern@gm ail.com> wrote:
>[color=green]
>>Ron Garret wrote:
>>[color=darkred]
>>>I'm using an OS X terminal to ssh to a Linux machine.[/color]
>>
>>Click on the "Terminal" menu, then "Window Settings...". Choose "Display"
>>from
>>the combobox. At the bottom you will see a combobox title "Character Set
>>Encoding". Choose "Unicode (UTF-8)".[/color]
>
> It was already set to UTF-8.[/color]

Then take a look at your LANG environment variable on your Linux machine. For
example, I have LANG=en_US.UTF-8 on my Linux machine, and I can ssh into it from
a UTF-8-configured Terminal.app and print unicode strings just fine.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Ron Garret]

Re: WTF? Printing unicode strings

In article <1147998139.268 318.315250@y43g 2000cwc.googleg roups.com>,
"Serge Orlov" <Serge.Orlov@gm ail.com> wrote:
[color=blue]
> Ron Garret wrote:[color=green]
> > In article <1147992722.970 761.220840@j73g 2000cwa.googleg roups.com>,
> > "Serge Orlov" <Serge.Orlov@gm ail.com> wrote:
> >[color=darkred]
> > > Ron Garret wrote:
> > > > In article <mailman.5906.1 147989402.27775 .python-list@python.org >,
> > > > Robert Kern <robert.kern@gm ail.com> wrote:
> > > >
> > > > > Ron Garret wrote:
> > > > >
> > > > > > I forgot to mention:
> > > > > >
> > > > > >>>>sys.getdefa ultencoding()
> > > > > >
> > > > > > 'utf-8'
> > > > >
> > > > > A) You shouldn't be able to do that.
> > > >
> > > > What can I say? I can.
> > > >
> > > > > B) Don't do that.
> > > >
> > > > OK. What should I do instead?
> > >
> > > Exact answer depends on what OS and terminal you are using and what
> > > your program is supposed to do, are you going to distribute the program
> > > or it's just for internal use.[/color]
> >
> > I'm using an OS X terminal to ssh to a Linux machine.[/color]
>
> In theory it should work out of the box. OS X terminal should set
> enviromental variable LANG=en_US.utf-8, then ssh should transfer this
> variable to Linux and python will know that your terminal is utf-8.
> Unfortunately AFAIK OS X terminal doesn't set that variable and most
> (all?) ssh clients don't transfer it between machines. As a workaround
> you can set that variable on linux yourself . This should work in the
> command line right away:
>
> LANG=en_US.utf-8 python -c "print unichr(0xbd)"
>
> Or put the following line in ~/.bashrc and logout/login
>
> export LANG=en_US.utf-8[/color]

No joy.

ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
Traceback (most recent call last):
File "<string>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)
ron@www01:~$

rg