Unicode conversion in 'print'

[Ricardo Bugalho]

Hello,
I'm using Python 2.3.4 and I noticed that, when stdout is a terminal, the
'print' statement converts Unicode strings into the encoding defined by
the locales instead of the one returned by sys.getdefaulte ncoding().
However, I can't find any references to it. Anyone knows where it's
descrbed?

Example:

!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys, locale

print 'Python encoding:', sys.getdefaulte ncoding()
print 'System encoding:', locale.getprefe rredencoding()
print 'Test string: ', u'Olá mundo'


If stdout is a terminal, works fine
$ python x.py
Python encoding: ascii
System encoding: UTF-8
Test string: Olá mundo

If I redirect the output to a file, raises an UnicodeEncodeEr ror exception
$ python x.py > x.txt
Traceback (most recent call last):
File "x.py", line 8, in ?
print 'Test string: ', u'Olá mundo'
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)


--
Ricardo

[Serge Orlov]

Re: Unicode conversion in 'print'

Ricardo Bugalho wrote:[color=blue]
> Hello,
> I'm using Python 2.3.4 and I noticed that, when stdout is a[/color]
terminal,[color=blue]
> the 'print' statement converts Unicode strings into the encoding
> defined by the locales instead of the one returned by
> sys.getdefaulte ncoding().[/color]

Sure. It uses the encoding of you console. Here is explanation why it
uses locale to get the encoding of console:

http://www.python.org/moin/PrintFails

[color=blue]
> However, I can't find any references to it. Anyone knows where it's
> descrbed?[/color]

I've just wrote about it here:

http://www.python.org/moin/DefaultEncoding

[color=blue]
>
> Example:
>
> !/usr/bin/env python
> # -*- coding: utf-8 -*-
>
> import sys, locale
>
> print 'Python encoding:', sys.getdefaulte ncoding()
> print 'System encoding:', locale.getprefe rredencoding()
> print 'Test string: ', u'Olá mundo'
>
>
> If stdout is a terminal, works fine
> $ python x.py
> Python encoding: ascii
> System encoding: UTF-8
> Test string: Olá mundo
>
> If I redirect the output to a file, raises an UnicodeEncodeEr ror[/color]
exception[color=blue]
> $ python x.py > x.txt
> Traceback (most recent call last):
> File "x.py", line 8, in ?
> print 'Test string: ', u'Olá mundo'
> UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xe1' in[/color]
position 2: ordinal not in range(128)[color=blue]
>[/color]

http://www.python.org/moin/ShellRedirectionFails

Feel free to reply here if something is not clear, corrections in wiki
are also welcome.

Serge.

[Ricardo Bugalho]

Re: Unicode conversion in 'print'

Hi,
thanks for the information. But what I was really looking for was
informaion on when and why Python started doing it (previously, it always
used sys.getdefaulte ncoding())) and why it was done only for 'print' when
stdout is a terminal instead of always.

On Thu, 13 Jan 2005 14:33:20 -0800, Serge Orlov wrote:
[color=blue]
> Sure. It uses the encoding of you console. Here is explanation why it uses
> locale to get the encoding of console:
> http://www.python.org/moin/PrintFails
>[/color]
--
Ricardo

[Serge Orlov]

Re: Unicode conversion in 'print'

Ricardo Bugalho wrote:[color=blue]
> Hi,
> thanks for the information. But what I was really looking for was
> informaion on when and why Python started doing it (previously, it
> always used sys.getdefaulte ncoding()))[/color]

I don't have access to any other version except 2.2 at the moment but I
believe it happened between 2.2 and 2.3 for Windows and UNIX terminals.
On other unsupported terminals I suspect sys.getdefaulte ncoding is
still used. The reason for the change is proper support of unicode
input/output.

[color=blue]
> and why it was done only for 'print' when
> stdout is a terminal instead of always.[/color]

The real question is why not *never* use sys.getdefaulte ncoding()
for printing. If you leave sys.getdefaulte ncoding() at Python default
value ('ascii') you won't need to worry about it <wink>
sys.getdefaulte ncoding() is a temporary measure for big projects to
use within one Python version.

Serge.

[Martin v. Löwis]

Re: Unicode conversion in 'print'

Ricardo Bugalho wrote:[color=blue]
> thanks for the information. But what I was really looking for was
> informaion on when and why Python started doing it (previously, it always
> used sys.getdefaulte ncoding())) and why it was done only for 'print' when
> stdout is a terminal instead of always.[/color]

It does that since 2.2, in response to many complains that you cannot
print a Unicode string in interactive mode, unless the Unicode string
contains only ASCII characters. It does that only if sys.stdout is
a real terminal, because otherwise it is not possible to determine
what the encoding of sys.stdout is.

Regards,
Martin