WTF? Printing unicode strings

**Serge Orlov** · May 19 '06, 01:25 AM

Re: WTF? Printing unicode strings

Ron Garret wrote:[color=blue][color=green][color=darkred]
> > > I'm using an OS X terminal to ssh to a Linux machine.[/color]
> >
> > In theory it should work out of the box. OS X terminal should set
> > enviromental variable LANG=en_US.utf-8, then ssh should transfer this
> > variable to Linux and python will know that your terminal is utf-8.
> > Unfortunately AFAIK OS X terminal doesn't set that variable and most
> > (all?) ssh clients don't transfer it between machines. As a workaround
> > you can set that variable on linux yourself . This should work in the
> > command line right away:
> >
> > LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> >
> > Or put the following line in ~/.bashrc and logout/login
> >
> > export LANG=en_US.utf-8[/color]
>
> No joy.
>
> ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> Traceback (most recent call last):
> File "<string>", line 1, in ?
> UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
> position 0: ordinal not in range(128)
> ron@www01:~$[/color]

What version of python and what shell do you run? What the following
commands print:

python -V
echo $SHELL
$SHELL --version

**Ron Garret** · May 19 '06, 07:05 AM

Re: WTF? Printing unicode strings

In article <1148001708.183 506.296240@g10g 2000cwb.googleg roups.com>,
"Serge Orlov" <Serge.Orlov@gm ail.com> wrote:
[color=blue]
> Ron Garret wrote:[color=green][color=darkred]
> > > > I'm using an OS X terminal to ssh to a Linux machine.
> > >
> > > In theory it should work out of the box. OS X terminal should set
> > > enviromental variable LANG=en_US.utf-8, then ssh should transfer this
> > > variable to Linux and python will know that your terminal is utf-8.
> > > Unfortunately AFAIK OS X terminal doesn't set that variable and most
> > > (all?) ssh clients don't transfer it between machines. As a workaround
> > > you can set that variable on linux yourself . This should work in the
> > > command line right away:
> > >
> > > LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > >
> > > Or put the following line in ~/.bashrc and logout/login
> > >
> > > export LANG=en_US.utf-8[/color]
> >
> > No joy.
> >
> > ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > Traceback (most recent call last):
> > File "<string>", line 1, in ?
> > UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
> > position 0: ordinal not in range(128)
> > ron@www01:~$[/color]
>
> What version of python and what shell do you run? What the following
> commands print:
>
> python -V
> echo $SHELL
> $SHELL --version[/color]

ron@www01:~$ python -V
Python 2.3.4
ron@www01:~$ echo $SHELL
/bin/bash
ron@www01:~$ $SHELL --version
GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
Copyright (C) 2002 Free Software Foundation, Inc.
ron@www01:~$

**Serge Orlov** · May 19 '06, 09:05 AM

Re: WTF? Printing unicode strings

Ron Garret wrote:[color=blue]
> In article <1148001708.183 506.296240@g10g 2000cwb.googleg roups.com>,
> "Serge Orlov" <Serge.Orlov@gm ail.com> wrote:
>[color=green]
> > Ron Garret wrote:[color=darkred]
> > > > > I'm using an OS X terminal to ssh to a Linux machine.
> > > >
> > > > In theory it should work out of the box. OS X terminal should set
> > > > enviromental variable LANG=en_US.utf-8, then ssh should transfer this
> > > > variable to Linux and python will know that your terminal is utf-8.
> > > > Unfortunately AFAIK OS X terminal doesn't set that variable and most
> > > > (all?) ssh clients don't transfer it between machines. As a workaround
> > > > you can set that variable on linux yourself . This should work in the
> > > > command line right away:
> > > >
> > > > LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > > >
> > > > Or put the following line in ~/.bashrc and logout/login
> > > >
> > > > export LANG=en_US.utf-8
> > >
> > > No joy.
> > >
> > > ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > > Traceback (most recent call last):
> > > File "<string>", line 1, in ?
> > > UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
> > > position 0: ordinal not in range(128)
> > > ron@www01:~$[/color]
> >
> > What version of python and what shell do you run? What the following
> > commands print:
> >
> > python -V
> > echo $SHELL
> > $SHELL --version[/color]
>
> ron@www01:~$ python -V
> Python 2.3.4
> ron@www01:~$ echo $SHELL
> /bin/bash
> ron@www01:~$ $SHELL --version
> GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
> Copyright (C) 2002 Free Software Foundation, Inc.
> ron@www01:~$[/color]

That's recent enough. I guess the distribution you're using set LC_*
variables for no good reason. Either unset all enviromental variables
starting with LC_ and set LANG variable or overide LC_CTYPE variable:

LC_CTYPE=en_US. utf-8 python -c "print unichr(0xbd)"

Should be working now :)

**Serge Orlov** · May 19 '06, 11:35 AM

Re: WTF? Printing unicode strings

Serge Orlov wrote:[color=blue]
> Ron Garret wrote:[color=green]
> > In article <1148001708.183 506.296240@g10g 2000cwb.googleg roups.com>,
> > "Serge Orlov" <Serge.Orlov@gm ail.com> wrote:
> >[color=darkred]
> > > Ron Garret wrote:
> > > > > > I'm using an OS X terminal to ssh to a Linux machine.
> > > > >
> > > > > In theory it should work out of the box. OS X terminal should set
> > > > > enviromental variable LANG=en_US.utf-8, then ssh should transfer this
> > > > > variable to Linux and python will know that your terminal is utf-8.
> > > > > Unfortunately AFAIK OS X terminal doesn't set that variable and most
> > > > > (all?) ssh clients don't transfer it between machines. As a workaround
> > > > > you can set that variable on linux yourself . This should work in the
> > > > > command line right away:
> > > > >
> > > > > LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > > > >
> > > > > Or put the following line in ~/.bashrc and logout/login
> > > > >
> > > > > export LANG=en_US.utf-8
> > > >
> > > > No joy.
> > > >
> > > > ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > > > Traceback (most recent call last):
> > > > File "<string>", line 1, in ?
> > > > UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
> > > > position 0: ordinal not in range(128)
> > > > ron@www01:~$
> > >
> > > What version of python and what shell do you run? What the following
> > > commands print:
> > >
> > > python -V
> > > echo $SHELL
> > > $SHELL --version[/color]
> >
> > ron@www01:~$ python -V
> > Python 2.3.4
> > ron@www01:~$ echo $SHELL
> > /bin/bash
> > ron@www01:~$ $SHELL --version
> > GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
> > Copyright (C) 2002 Free Software Foundation, Inc.
> > ron@www01:~$[/color]
>
> That's recent enough. I guess the distribution you're using set LC_*
> variables for no good reason. Either unset all enviromental variables
> starting with LC_ and set LANG variable or overide LC_CTYPE variable:
>
> LC_CTYPE=en_US. utf-8 python -c "print unichr(0xbd)"
>
> Should be working now :)[/color]

I've pulled myself together and installed linux in vwware player.
Apparently there is another way linux distributors can screw up. I
chose debian 3.1 minimal network install and after answering all
installation questions I found that only ascii and latin-1 english
locales were installed:
$ locale -a
C
en_US
en_US.iso88591
POSIX

In 2006, I would expect utf-8 english locale to be present even in
minimal install. I had to edit /etc/locale.gen and run locale-gen as
root. After that python started to print unicode characters.

**Laurent Pointal** · May 19 '06, 12:15 PM

Re: WTF? Printing unicode strings

Ron Garret a écrit :[color=blue]
> In article <mailman.5904.1 147987926.27775 .python-list@python.org >,
> Fredrik Lundh <fredrik@python ware.com> wrote:
>[color=green]
>> Ron Garret wrote:
>>[color=darkred]
>>>>>> u'\xbd'
>>> u'\xbd'
>>>>>> print _
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in ?
>>> UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
>>> position 0: ordinal not in range(128)[/color]
>> so stdout on your machine is ascii, and you don't understand why you
>> cannot print a non-ascii unicode character to it? wtf?
>>
>> </F>[/color]
>
> I forgot to mention:
>[color=green][color=darkred]
>>>> sys.getdefaulte ncoding()[/color][/color]
> 'utf-8'[color=green][color=darkred]
>>>> print u'\xbd'[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
> position 0: ordinal not in range(128)[/color]

This is default encoding for evaluation of expressions in u"..."
strings, this has nothing to do with printing.

For the output encoding, see sys.stdout.enco ding.
[color=blue][color=green][color=darkred]
>>> import sys
>>> sys.stdout.enco ding[/color][/color][/color]
'cp850'[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

A+

Laurent.

**John Salerno** · May 19 '06, 02:35 PM

Re: WTF? Printing unicode strings

Fredrik Lundh wrote:[color=blue]
> Ron Garret wrote:
>[color=green][color=darkred]
>>>>> u'\xbd'[/color]
>> u'\xbd'[color=darkred]
>>>>> print _[/color]
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in ?
>> UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
>> position 0: ordinal not in range(128)[/color]
>
> so stdout on your machine is ascii, and you don't understand why you
> cannot print a non-ascii unicode character to it? wtf?
>
> </F>
>[/color]

AFAIK, I'm all ASCII (at least, I never made explicit changes to the
default Python install), so how am I able to print out the character?

**Robert Kern** · May 19 '06, 04:35 PM

Re: WTF? Printing unicode strings

John Salerno wrote:
[color=blue]
> AFAIK, I'm all ASCII (at least, I never made explicit changes to the
> default Python install), so how am I able to print out the character?[/color]

Because sys.stdout.enco ding isn't determined by your Python configuration, but
your terminal's.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

**skip@pobox.com** · May 19 '06, 05:15 PM

Re: WTF? Printing unicode strings

Robert> Because sys.stdout.enco ding isn't determined by your Python
Robert> configuration, but your terminal's.

Learn something every day. I take it "646" is an alias for "ascii" (or vice
versa)?

% python
Python 2.4.2 (#1, Feb 23 2006, 12:48:31)
[GCC 3.4.1] on sunos5
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> import sys
>>> sys.stdout.enco ding[/color][/color][/color]
'646'[color=blue][color=green][color=darkred]
>>> import codecs
>>> codecs.lookup(" 646")[/color][/color][/color]
(<built-in function ascii_encode>, <built-in function ascii_decode>, <class encodings.ascii .StreamReader at 0x819aa4c>, <class encodings.ascii .StreamWriter at 0x819aa1c>)

Skip

**John Salerno** · May 19 '06, 05:35 PM

Re: WTF? Printing unicode strings

skip@pobox.com wrote:[color=blue]
> Robert> Because sys.stdout.enco ding isn't determined by your Python
> Robert> configuration, but your terminal's.
>
> Learn something every day. I take it "646" is an alias for "ascii" (or vice
> versa)?[/color]

Hmm, not that this helps me any :)
[color=blue][color=green][color=darkred]
>>> import sys
>>> sys.stdout.enco ding[/color][/color][/color]
'cp1252'[color=blue][color=green][color=darkred]
>>> import codecs
>>> codecs.lookup(' cp1252')[/color][/color][/color]
(<bound method Codec.encode of <encodings.cp12 52.Codec instance at
0x009D6670>>, <bound method Codec.decode of <encodings.cp12 52.Codec
instance at 0x009D6698>>, <class encodings.cp125 2.StreamReader at
0x009CF360>, <class encodings.cp125 2.StreamWriter at 0x009CF330>)[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

**skip@pobox.com** · May 19 '06, 05:45 PM

Re: WTF? Printing unicode strings

John> Hmm, not that this helps me any :)
[color=blue][color=green][color=darkred]
>>>> import sys
>>>> sys.stdout.enco ding[/color][/color][/color]
John> 'cp1252'

Sure it does. You can print Unicode objects which map to cp1252. I assume
that means you're on Windows or that for some perverse reason you have your
Mac's Terminal window set to cp1252. (Does it go there? I'm at work right
now so I can't check).

Skip

**John Salerno** · May 19 '06, 05:45 PM

Re: WTF? Printing unicode strings

skip@pobox.com wrote:[color=blue]
> John> Hmm, not that this helps me any :)
>[color=green][color=darkred]
> >>>> import sys
> >>>> sys.stdout.enco ding[/color][/color]
> John> 'cp1252'
>
> Sure it does. You can print Unicode objects which map to cp1252. I assume
> that means you're on Windows or that for some perverse reason you have your
> Mac's Terminal window set to cp1252. (Does it go there? I'm at work right
> now so I can't check).
>
> Skip[/color]

You're right, I'm on XP. I just couldn't make sense of the lookup call,
although some of the names looked like .NET classes.

**Robert Kern** · May 19 '06, 06:15 PM

Re: WTF? Printing unicode strings

skip@pobox.com wrote:[color=blue]
> Robert> Because sys.stdout.enco ding isn't determined by your Python
> Robert> configuration, but your terminal's.
>
> Learn something every day. I take it "646" is an alias for "ascii" (or vice
> versa)?
>
> % python
> Python 2.4.2 (#1, Feb 23 2006, 12:48:31)
> [GCC 3.4.1] on sunos5
> Type "help", "copyright" , "credits" or "license" for more information.[color=green][color=darkred]
> >>> import sys
> >>> sys.stdout.enco ding[/color][/color]
> '646'[color=green][color=darkred]
> >>> import codecs
> >>> codecs.lookup(" 646")[/color][/color]
> (<built-in function ascii_encode>, <built-in function ascii_decode>, <class encodings.ascii .StreamReader at 0x819aa4c>, <class encodings.ascii .StreamWriter at 0x819aa1c>)[/color]

Yes. In encodings/aliases.py in the standard library:

"""
aliases = {

# Please keep this list sorted alphabetically by value !

# ascii codec
'646' : 'ascii',

"""

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

**Ron Garret** · May 19 '06, 10:55 PM

Re: WTF? Printing unicode strings

In article <1148029074.083 644.112370@i40g 2000cwc.googleg roups.com>,
"Serge Orlov" <Serge.Orlov@gm ail.com> wrote:
[color=blue]
> Ron Garret wrote:[color=green]
> > In article <1148001708.183 506.296240@g10g 2000cwb.googleg roups.com>,
> > "Serge Orlov" <Serge.Orlov@gm ail.com> wrote:
> >[color=darkred]
> > > Ron Garret wrote:
> > > > > > I'm using an OS X terminal to ssh to a Linux machine.
> > > > >
> > > > > In theory it should work out of the box. OS X terminal should set
> > > > > enviromental variable LANG=en_US.utf-8, then ssh should transfer this
> > > > > variable to Linux and python will know that your terminal is utf-8.
> > > > > Unfortunately AFAIK OS X terminal doesn't set that variable and most
> > > > > (all?) ssh clients don't transfer it between machines. As a workaround
> > > > > you can set that variable on linux yourself . This should work in the
> > > > > command line right away:
> > > > >
> > > > > LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > > > >
> > > > > Or put the following line in ~/.bashrc and logout/login
> > > > >
> > > > > export LANG=en_US.utf-8
> > > >
> > > > No joy.
> > > >
> > > > ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > > > Traceback (most recent call last):
> > > > File "<string>", line 1, in ?
> > > > UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
> > > > position 0: ordinal not in range(128)
> > > > ron@www01:~$
> > >
> > > What version of python and what shell do you run? What the following
> > > commands print:
> > >
> > > python -V
> > > echo $SHELL
> > > $SHELL --version[/color]
> >
> > ron@www01:~$ python -V
> > Python 2.3.4
> > ron@www01:~$ echo $SHELL
> > /bin/bash
> > ron@www01:~$ $SHELL --version
> > GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
> > Copyright (C) 2002 Free Software Foundation, Inc.
> > ron@www01:~$[/color]
>
> That's recent enough. I guess the distribution you're using set LC_*
> variables for no good reason.[/color]

Nope:

ron@www01:~$ export | grep LC
ron@www01:~$
[color=blue]
> Either unset all enviromental variables
> starting with LC_ and set LANG variable or overide LC_CTYPE variable:
>
> LC_CTYPE=en_US. utf-8 python -c "print unichr(0xbd)"
>
> Should be working now :)[/color]

Nope:

ron@www01:~$ LC_CTYPE=en_US. utf-8 python -c "print unichr(0xbd)"
Traceback (most recent call last):
File "<string>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)

rg

**Ron Garret** · May 19 '06, 10:55 PM

Re: WTF? Printing unicode strings

In article <1148038129.231 027.118010@g10g 2000cwb.googleg roups.com>,
"Serge Orlov" <Serge.Orlov@gm ail.com> wrote:
[color=blue]
> Serge Orlov wrote:[color=green]
> > Ron Garret wrote:[color=darkred]
> > > In article <1148001708.183 506.296240@g10g 2000cwb.googleg roups.com>,
> > > "Serge Orlov" <Serge.Orlov@gm ail.com> wrote:
> > >
> > > > Ron Garret wrote:
> > > > > > > I'm using an OS X terminal to ssh to a Linux machine.
> > > > > >
> > > > > > In theory it should work out of the box. OS X terminal should set
> > > > > > enviromental variable LANG=en_US.utf-8, then ssh should transfer
> > > > > > this
> > > > > > variable to Linux and python will know that your terminal is utf-8.
> > > > > > Unfortunately AFAIK OS X terminal doesn't set that variable and
> > > > > > most
> > > > > > (all?) ssh clients don't transfer it between machines. As a
> > > > > > workaround
> > > > > > you can set that variable on linux yourself . This should work in
> > > > > > the
> > > > > > command line right away:
> > > > > >
> > > > > > LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > > > > >
> > > > > > Or put the following line in ~/.bashrc and logout/login
> > > > > >
> > > > > > export LANG=en_US.utf-8
> > > > >
> > > > > No joy.
> > > > >
> > > > > ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > > > > Traceback (most recent call last):
> > > > > File "<string>", line 1, in ?
> > > > > UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xbd' in
> > > > > position 0: ordinal not in range(128)
> > > > > ron@www01:~$
> > > >
> > > > What version of python and what shell do you run? What the following
> > > > commands print:
> > > >
> > > > python -V
> > > > echo $SHELL
> > > > $SHELL --version
> > >
> > > ron@www01:~$ python -V
> > > Python 2.3.4
> > > ron@www01:~$ echo $SHELL
> > > /bin/bash
> > > ron@www01:~$ $SHELL --version
> > > GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
> > > Copyright (C) 2002 Free Software Foundation, Inc.
> > > ron@www01:~$[/color]
> >
> > That's recent enough. I guess the distribution you're using set LC_*
> > variables for no good reason. Either unset all enviromental variables
> > starting with LC_ and set LANG variable or overide LC_CTYPE variable:
> >
> > LC_CTYPE=en_US. utf-8 python -c "print unichr(0xbd)"
> >
> > Should be working now :)[/color]
>
> I've pulled myself together and installed linux in vwware player.
> Apparently there is another way linux distributors can screw up. I
> chose debian 3.1 minimal network install and after answering all
> installation questions I found that only ascii and latin-1 english
> locales were installed:
> $ locale -a
> C
> en_US
> en_US.iso88591
> POSIX
>
> In 2006, I would expect utf-8 english locale to be present even in
> minimal install. I had to edit /etc/locale.gen and run locale-gen as
> root. After that python started to print unicode characters.[/color]

That's it. Thanks!

rg

**Martin v. Löwis** · May 21 '06, 10:15 AM

Re: WTF? Printing unicode strings

skip@pobox.com wrote:[color=blue]
> Learn something every day. I take it "646" is an alias for "ascii" (or vice
> versa)?[/color]

Usage of "646" as an alias for ASCII is primarily a Sun invention. When
ASCII became an international standard, its standard number became
ISO/IEC 646:1968. It's not *quite* the same as ASCII, as it leaves a
certain number of code points unassigned that ASCII defines (most
notably, the dollar sign, and the square and curly braces). What Sun
means is probably the "Internatio nal Reference Version" of ISO 646,
which is (now) identical to ASCII.

Regards,
Martin

WTF? Printing unicode strings

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment