unicode question

[wolfgang haefelinger]

Hi,

I wonder whether someone could explain me a bit what's going on here:

import sys

# I'm running Mandrake 1o and Windows XP.
print sys.version

## 2.3.3 (#2, Feb 17 2004, 11:45:40) [GCC 3.3.2 (Mandrake Linux 10.0
3.3.2-6mdk)]
## 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)]

print "sys.getdefault encoding = ",sys.getdefaul tencoding()
# This prints always "ascii" ..

## just a class
class Y:
def __str__(self):
return self.c

## define unicode character (ie. string)
gamma = u"\N{GREEK CAPITAL LETTER GAMMA}"

y = Y()
y.c = gamma

## works fine: prints greek capital gamma on terminal on windows (chcp 437).
## Mandrake 1o nothing gets printed but at least no excecption gets thrown.
print gamma # (1)

## same as before ..
print y.__str__() # (2)

## encoding error
print y # (3) ??????????????

## ascii encoding error ..
sys.stdout.writ e(gamma) # (4)

I wonder especially about case 2. I can see that "print y" makes a call to
Y.__str__() . But Y.__str__() can be printed?? So what is 'print' exactly
doing?

Thanks for any help,
Wolfgang.

[Martin v. Löwis]

Re: unicode question

wolfgang haefelinger wrote:[color=blue]
> I wonder especially about case 2. I can see that "print y" makes a call to
> Y.__str__() . But Y.__str__() can be printed?? So what is 'print' exactly
> doing?[/color]

It looks at sys.stdout.enco ding. If this is set, and the thing to print
is a unicode string, it converts it to the stream encoding, and prints
the result of the conversion.

Regards,
Martin

[Kent Johnson]

Re: unicode question

Martin v. Löwis wrote:[color=blue]
> wolfgang haefelinger wrote:
>[color=green]
>> I wonder especially about case 2. I can see that "print y" makes a
>> call to
>> Y.__str__() . But Y.__str__() can be printed?? So what is 'print' exactly
>> doing?[/color]
>
>
> It looks at sys.stdout.enco ding. If this is set, and the thing to print
> is a unicode string, it converts it to the stream encoding, and prints
> the result of the conversion.[/color]

I hate to contradict an expert, but ISTM that it is
sys.getdefaulte ncoding() ('ascii') that is the problem, not
sys.stdout.enco ding ('cp437')

gamma converts to cp437 just fine:[color=blue][color=green][color=darkred]
>>> gamma = u"\N{GREEK CAPITAL LETTER GAMMA}"
>>> sys.stdout.enco ding[/color][/color][/color]
'cp437'[color=blue][color=green][color=darkred]
>>> gamma.encode(sy s.stdout.encodi ng)[/color][/color][/color]
'\xe2'[color=blue][color=green][color=darkred]
>>> print gamma.encode(sy s.stdout.encodi ng)[/color][/color][/color]
Γ
(prints a gamma)

Trying to encode gamma using the 'ascii' codec doesn't work:[color=blue][color=green][color=darkred]
>>> str(gamma)[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\u0393' in
position 0: ordinal not in range(128)

My guess is that internally, print keeps calling str() on its argument
until it gets a string object. So it calls y.__str__() yielding gamma,
then gamma.__str__() which raises the error.

If the default encoding is set to cp437 then it works fine:
[color=blue][color=green][color=darkred]
>>> import sys
>>> sys.getdefaulte ncoding()[/color][/color][/color]
'cp437'[color=blue][color=green][color=darkred]
>>> gamma = u"\N{GREEK CAPITAL LETTER GAMMA}"
>>> str(gamma)[/color][/color][/color]
'\xe2'[color=blue][color=green][color=darkred]
>>> print gamma[/color][/color][/color]
Γ
(prints a gamma)
[color=blue][color=green][color=darkred]
>>> print str(gamma)[/color][/color][/color]
Γ
(prints a gamma)

Kent
[color=blue]
>
> Regards,
> Martin[/color]

[Martin v. Löwis]

Re: unicode question

Kent Johnson wrote:[color=blue]
> Martin v. Löwis wrote:
>[color=green]
>> wolfgang haefelinger wrote:
>>[color=darkred]
>>> I wonder especially about case 2. I can see that "print y" makes a
>>> call to
>>> Y.__str__() . But Y.__str__() can be printed?? So what is 'print'
>>> exactly doing?[/color]
>>
>>
>>
>> It looks at sys.stdout.enco ding. If this is set, and the thing to print
>> is a unicode string, it converts it to the stream encoding, and prints
>> the result of the conversion.[/color]
>
>
> I hate to contradict an expert, but ISTM that it is
> sys.getdefaulte ncoding() ('ascii') that is the problem, not
> sys.stdout.enco ding ('cp437')[/color]

It seems we were answering different parts of the question. I answered
the part "What is 'print' exactly doing"; you answered the part as to
what the problem with str() conversion is (although I'm not sure whether
the OP has actually asked that question).

Also, the one case that is interesting here was not in your experiment:
try

print gamma

This should work, regardless of sys.getdefaulte ncoding(), as long as
sys.stdout.enco ding supports the characters to be printed.

Regards,
Martin

[wolfgang haefelinger]

Re: unicode question

Hi Experts,

I'm actually not a Python expert so please bear with me and my naive
questions and remarks:

I was actually thinking that

print x

is just kind of shortcur for writing (simplifying bit):

import sys
if not (isinstance(x,s tr) or isinstance(x,un icode)) and x.__str__ :
x = x.__str__()
sys.stdout.writ e(x)

Or in words: if x is not a string type but has method __str__ then

print x

behaves like

print x.__str__()

Given this assumption I'm wondering then why print x.__str__()
works but print x does not?

Is this a bug??

Cheers,
Wolfgang.



""Martin v. Löwis"" <martin@v.loewi s.de> wrote in message
news:41A061E1.8 050203@v.loewis .de...[color=blue]
> Kent Johnson wrote:[color=green]
>> Martin v. Löwis wrote:
>>[color=darkred]
>>> wolfgang haefelinger wrote:
>>>
>>>> I wonder especially about case 2. I can see that "print y" makes a call
>>>> to
>>>> Y.__str__() . But Y.__str__() can be printed?? So what is 'print'
>>>> exactly doing?
>>>
>>>
>>>
>>> It looks at sys.stdout.enco ding. If this is set, and the thing to print
>>> is a unicode string, it converts it to the stream encoding, and prints
>>> the result of the conversion.[/color]
>>
>>
>> I hate to contradict an expert, but ISTM that it is
>> sys.getdefaulte ncoding() ('ascii') that is the problem, not
>> sys.stdout.enco ding ('cp437')[/color]
>
> It seems we were answering different parts of the question. I answered
> the part "What is 'print' exactly doing"; you answered the part as to
> what the problem with str() conversion is (although I'm not sure whether
> the OP has actually asked that question).
>
> Also, the one case that is interesting here was not in your experiment:
> try
>
> print gamma
>
> This should work, regardless of sys.getdefaulte ncoding(), as long as
> sys.stdout.enco ding supports the characters to be printed.
>
> Regards,
> Martin[/color]

[Martin v. Löwis]

Re: unicode question

wolfgang haefelinger wrote:[color=blue]
> I was actually thinking that
>
> print x
>
> is just kind of shortcur for writing (simplifying bit):
>
> import sys
> if not (isinstance(x,s tr) or isinstance(x,un icode)) and x.__str__ :
> x = x.__str__()
> sys.stdout.writ e(x)[/color]

This is too simplifying. For the context of this discussion,
it is rather

import sys
if isinstance(x, unicode) and sys.stdout.enco ding:
x = x.encode(sys.st dout.encoding)
x = str(x)
sys.stdout.writ e(x)

(this, of course, is still quite simplicated. It ignores tp_print,
and it ignores softspaces).
[color=blue]
> Or in words: if x is not a string type but has method __str__ then
>
> print x
>
> behaves like
>
> print x.__str__()[/color]

No. There are many types for which this is not true; in this specific
case, it isn't true for Unicode objects.
[color=blue]
> Is this a bug??[/color]

No. You are just misunderstandin g it.

Regards,
Martin

[wolfgang haefelinger]

Re: unicode question

Hi Martin,

if print is implemented like this then I begin to understand the problem.

Neverthelss, I regard

print y.__str__() ## works
print y ## fails??

as a very inconsistent behaviour.

Somehow I have the feeling that Python should give up the distinction
between unicode and str and just have a str type which is internally
unicode.


Anyway, thanks for answering
Wolfgang.

""Martin v. Löwis"" <martin@v.loewi s.de> wrote in message
news:41a0ab62$0 $151$9b622d9e@n ews.freenet.de. ..[color=blue]
> wolfgang haefelinger wrote:[color=green]
>> I was actually thinking that
>>
>> print x
>>
>> is just kind of shortcur for writing (simplifying bit):
>>
>> import sys
>> if not (isinstance(x,s tr) or isinstance(x,un icode)) and x.__str__ :
>> x = x.__str__()
>> sys.stdout.writ e(x)[/color]
>
> This is too simplifying. For the context of this discussion,
> it is rather
>
> import sys
> if isinstance(x, unicode) and sys.stdout.enco ding:
> x = x.encode(sys.st dout.encoding)
> x = str(x)
> sys.stdout.writ e(x)
>
> (this, of course, is still quite simplicated. It ignores tp_print,
> and it ignores softspaces).
>[color=green]
>> Or in words: if x is not a string type but has method __str__ then
>>
>> print x
>>
>> behaves like
>>
>> print x.__str__()[/color]
>
> No. There are many types for which this is not true; in this specific
> case, it isn't true for Unicode objects.
>[color=green]
>> Is this a bug??[/color]
>
> No. You are just misunderstandin g it.
>
> Regards,
> Martin[/color]

[Bengt Richter]

Re: unicode question









On Mon, 22 Nov 2004 08:04:08 GMT, "wolfgang haefelinger" <wh2005@web.d e> wrote:
[color=blue]
>Hi Martin,
>
>if print is implemented like this then I begin to understand the problem.
>
>Neverthelss, I regard
>
> print y.__str__() ## works
> print y ## fails??
>
>as a very inconsistent behaviour.
>
>Somehow I have the feeling that Python should give up the distinction
>between unicode and str and just have a str type which is internally
>unicode.
>
>
>Anyway, thanks for answering
>Wolfgang.
>
>""Martin v. Löwis"" <martin@v.loewi s.de> wrote in message
>news:41a0ab62$ 0$151$9b622d9e@ news.freenet.de ...[color=green]
>> wolfgang haefelinger wrote:[color=darkred]
>>> I was actually thinking that
>>>
>>> print x
>>>
>>> is just kind of shortcur for writing (simplifying bit):
>>>
>>> import sys
>>> if not (isinstance(x,s tr) or isinstance(x,un icode)) and x.__str__ :
>>> x = x.__str__()
>>> sys.stdout.writ e(x)[/color]
>>
>> This is too simplifying. For the context of this discussion,
>> it is rather
>>
>> import sys
>> if isinstance(x, unicode) and sys.stdout.enco ding:
>> x = x.encode(sys.st dout.encoding)
>> x = str(x)
>> sys.stdout.writ e(x)
>>
>> (this, of course, is still quite simplicated. It ignores tp_print,
>> and it ignores softspaces).
>>[color=darkred]
>>> Or in words: if x is not a string type but has method __str__ then
>>>
>>> print x
>>>
>>> behaves like
>>>
>>> print x.__str__()[/color]
>>
>> No. There are many types for which this is not true; in this specific
>> case, it isn't true for Unicode objects.
>>[color=darkred]
>>> Is this a bug??[/color]
>>
>> No. You are just misunderstandin g it.
>>
>> Regards,
>> Martin[/color]
>[/color]
It's an old issue, and ISTM there is either a problem or it needs to be better explained.
My bet is on a problem ;-) ISTM the key is that a plain str type is a byte sequence but can
be interpreted as a byte-stream-encoded character sequence, and there are some seemingly
schizophrenic situations. E.g., start with a sequence of numbers, obviously just produced
by a polynomial formula having nothing to do with characters:
[color=blue][color=green][color=darkred]
>>> numbers = [(lambda x: (-499*x**4 +4634*x**3 -13973*x**2 +13918*x +1824)/24)(x) for x in xrange(5)]
>>> numbers[/color][/color][/color]
[76, 246, 119, 105, 115]

Now if we convert those to str type characters with chr() and join them:
[color=blue][color=green][color=darkred]
>>> s = ''.join(map(chr , numbers))[/color][/color][/color]

Then we have a sequence of bytes which could have had any numerical value in range(256). No character
encoding is assumed. Yet. If we now assume, say, a latin-1 encoding, we can decode the bytes into
unicode:
[color=blue][color=green][color=darkred]
>>> u = s.decode('latin-1')
>>> type(u)[/color][/color][/color]
<type 'unicode'>

Now if we print that, sys.stdout.enco ding should come into play:
[color=blue][color=green][color=darkred]
>>> print u[/color][/color][/color]
Löwis

:-)

And we are ok, because we were explicit the whole way.
But if we don't decode s explicitly, it seems the system makes an assumption:
[color=blue][color=green][color=darkred]
>>> print s[/color][/color][/color]
L÷wis

That is (if it survived) the 'cp437' character for byte '\xf6'. IOW, print seems
to assume that a plain str is encoded ready for output in sys.stdout.enco ding in
a kind of reinterpret_cas t of the str, or else a decode('cp437') .encode('cp437' )
optimized away.
[color=blue][color=green][color=darkred]
>>> sys.stdout.enco ding[/color][/color][/color]
'cp437'[color=blue][color=green][color=darkred]
>>> sys.getdefaulte ncoding()[/color][/color][/color]
'ascii'

If it were assuming s was encoded as ascii, it should really do s.decode('ascii ').encode('cp43 7')
to get it printed, but for plain str literals it does not seem to do that. I.e.,
[color=blue][color=green][color=darkred]
>>> s.decode('ascii ')[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xf6 in position 1: ordinal not in range(128

doesn't work, so it can't be doing that. It seems to print s as s.decode('cp437 ').encode('cp43 7')
[color=blue][color=green][color=darkred]
>>> s.decode('cp437 ')[/color][/color][/color]
u'L\xf7wis'

but that is a wrong decoding, (though the system can't be expected to know).
[color=blue][color=green][color=darkred]
>>> print s.decode('cp437 ').encode('cp43 7')[/color][/color][/color]
L÷wis[color=blue][color=green][color=darkred]
>>> print s.decode('latin-1').encode('cp4 37')[/color][/color][/color]
Löwis

What other decoding should be attempted, lacking an indication? sys.getdefaulte ncoding()
might be reasonable, but it seems to be locked into 'ascii' (I don't know how to set it)
[color=blue][color=green][color=darkred]
>>> sys.getdefaulte ncoding = lambda: 'latin-1'
>>> sys.getdefaulte ncoding()[/color][/color][/color]
'latin-1'[color=blue][color=green][color=darkred]
>>> unicode('L\xf6w is')[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xf6 in position 1: ordinal not in range(128


So, bottom line, as Wolfgang effectively asked by his example, why does print try to coerce
the __str__ return value to ascii on the way to the ouput encoder, when there is encoding info
in the unicode object that it is happy to defer reencoding of for sys.stdout.enco ding?
[color=blue][color=green][color=darkred]
>>> s[/color][/color][/color]
'L\xf6wis'[color=blue][color=green][color=darkred]
>>> u[/color][/color][/color]
u'L\xf6wis'[color=blue][color=green][color=darkred]
>>> print s[/color][/color][/color]
L÷wis[color=blue][color=green][color=darkred]
>>> print u[/color][/color][/color]
Löwis[color=blue][color=green][color=darkred]
>>> class Y:[/color][/color][/color]
... def __str__(self): return self.c
...[color=blue][color=green][color=darkred]
>>> y = Y()
>>> y.c = s
>>> print y[/color][/color][/color]
L÷wis[color=blue][color=green][color=darkred]
>>> y.c = u
>>> print y[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xf6' in position 1: ordinal not in r
ange(128)[color=blue][color=green][color=darkred]
>>> print u[/color][/color][/color]
Löwis

Maybe the output of __str__ should be ok as a type basestring subclass for print, so
y.c = u
print y
above has the same result as
print u

It seems to be trying to do u.encode('ascii ').decode('asci i').encode('cp4 37')
instead of directly u.encode('cp437 ') when __str__ is involved.
[color=blue][color=green][color=darkred]
>>> print u'%s' % y[/color][/color][/color]
Löwis

works, and
[color=blue][color=green][color=darkred]
>>> print '%s' % u[/color][/color][/color]
Löwis

works, and
[color=blue][color=green][color=darkred]
>>> print y.__str__()[/color][/color][/color]
Löwis

and
[color=blue][color=green][color=darkred]
>>> print y.c[/color][/color][/color]
Löwis

works,[color=blue][color=green][color=darkred]
>>> y.c[/color][/color][/color]
u'L\xf6wis'

but
[color=blue][color=green][color=darkred]
>>> print '%s'%y[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xf6' in position 1: ordinal not in r
ange(128)

and never mind print,
[color=blue][color=green][color=darkred]
>>> '%s' % u[/color][/color][/color]
u'L\xf6wis'[color=blue][color=green][color=darkred]
>>> '%s' % y.__str__()[/color][/color][/color]
u'L\xf6wis'[color=blue][color=green][color=darkred]
>>> '%s' % y[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xf6' in position 1: ordinal not in r
ange(128)

I guess its that str.__mod__(sel f, other) can deal with a unicode other and get promoted, but
it must do str(other) instead of other.__str__() , or it would be able to promote the result in
the latter case too...

This seems like a possible change that could smooth things a bit, especially if print a,b,c
was then effectively the same as print ('%s'%a),('%s'% b),('%s'%c) with encoding promotion.

Regards,
Bengt Richter

[Martin v. Löwis]

Re: unicode question

wolfgang haefelinger wrote:[color=blue]
> Neverthelss, I regard
>
> print y.__str__() ## works
> print y ## fails??
>
> as a very inconsistent behaviour.[/color]

Notice that this also fails

x=str(y)

So it is really the string conversion that fails. Roughly the same
happens with

class X:
def __str__(self):
return -1

Here, instances of X also cannot be printed: str() is really supposed
to return a byte string object - not a number, not a unicode object.
As a special exception, __str__ can return a Unicode object, as long
as that result can be converted with the system default encoding into
a byte string object. So we really have

def str(o):
if isinstance(o, types.StringTyp e): return o
if isinstance(o, types.UnicodeTy pe): return o.encode(None)
return str(o.__str__() )

This is why the first print succeeds (it calls __str__ directly,
printing the Unicode object afterwards), and the second print fails
(trying to str()-convert its argument, which already fails - it
didn't get so far as to actually trying to print something).
[color=blue]
> Somehow I have the feeling that Python should give up the distinction
> between unicode and str and just have a str type which is internally
> unicode.[/color]

Yes, that should happen in P3k. But even then, there will be a
distinction between byte (plain) strings, and character (unicode)
strings.

Regards,
Martin

[Martin v. Löwis]

Re: unicode question

Bengt Richter wrote:[color=blue]
> So, bottom line, as Wolfgang effectively asked by his example, why does print try to coerce
> the __str__ return value to ascii on the way to the ouput encoder, when there is encoding info
> in the unicode object that it is happy to defer reencoding of for sys.stdout.enco ding?[/color]

[See my other posting:]
Because print invokes str() on its argument, unless the argument is
already a byte string (in which case it prints it directly), or a
Unicode string (in which case it encodes it with the stream encoding).
It is str(y) that fails, not the printing.

Regards,
Martin

[Bengt Richter]

Re: unicode question

On Tue, 23 Nov 2004 00:24:09 +0100, =?ISO-8859-1?Q?=22Martin_v =2E_L=F6wis=22? = <martin@v.loewi s.de> wrote:
[color=blue]
>Bengt Richter wrote:[color=green]
>> So, bottom line, as Wolfgang effectively asked by his example, why does print try to coerce
>> the __str__ return value to ascii on the way to the ouput encoder, when there is encoding info
>> in the unicode object that it is happy to defer reencoding of for sys.stdout.enco ding?[/color]
>
>[See my other posting:]
>Because print invokes str() on its argument, unless the argument is
>already a byte string (in which case it prints it directly), or a[/color]
^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^ ^^^^^^-- effectively an assumption that
bytestring.deco de('some_unknow n_encoding').en code(sys.stdout .encoding)
has already been done, it seems (I'm not arguing against).
[color=blue]
>Unicode string (in which case it encodes it with the stream encoding).
>It is str(y) that fails, not the printing.
>[/color]
Yes, I think my turgid post did demonstrate that, among other things ;-)

So how about changing print so that it doesn't blindly use str(y), but instead
first tries to get y.__str__() in case the latter returns unicode?
Then print y can succeed the way print y.__str__() does now.

The same goes for str.__mod__ -- it apparently knows how to deal with '%s'% unicode(y)
so why shouldn't '%s'%y benefit when y.__str__ returns unicode?

I.e., str doesn't know that printing and '%s' can use unicode to good effect
if it available, so for print and str.__mod__ blindly to use str() intermediately
throws away an opportunity to do better ISTM.

Regards,
Bengt Richter

[Steve Holden]

Re: unicode question

Bengt Richter wrote:
[color=blue]
> On Tue, 23 Nov 2004 00:24:09 +0100, =?ISO-8859-1?Q?=22Martin_v =2E_L=F6wis=22? = <martin@v.loewi s.de> wrote:
>
>[color=green]
>>Bengt Richter wrote:
>>[color=darkred]
>>>So, bottom line, as Wolfgang effectively asked by his example, why does print try to coerce
>>>the __str__ return value to ascii on the way to the ouput encoder, when there is encoding info
>>>in the unicode object that it is happy to defer reencoding of for sys.stdout.enco ding?[/color]
>>
>>[See my other posting:]
>>Because print invokes str() on its argument, unless the argument is
>>already a byte string (in which case it prints it directly), or a[/color]
>
> ^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^ ^^^^^^-- effectively an assumption that
> bytestring.deco de('some_unknow n_encoding').en code(sys.stdout .encoding)
> has already been done, it seems (I'm not arguing against).
>
>[color=green]
>>Unicode string (in which case it encodes it with the stream encoding).
>>It is str(y) that fails, not the printing.
>>[/color]
>
> Yes, I think my turgid post did demonstrate that, among other things ;-)
>
> So how about changing print so that it doesn't blindly use str(y), but instead
> first tries to get y.__str__() in case the latter returns unicode?
> Then print y can succeed the way print y.__str__() does now.
>
> The same goes for str.__mod__ -- it apparently knows how to deal with '%s'% unicode(y)
> so why shouldn't '%s'%y benefit when y.__str__ returns unicode?
>
> I.e., str doesn't know that printing and '%s' can use unicode to good effect
> if it available, so for print and str.__mod__ blindly to use str() intermediately
> throws away an opportunity to do better ISTM.
>
> Regards,
> Bengt Richter[/color]

Am I the only person who found it scary that Bengt could apparently
casually drop on a polynomial the would decode to " Löwis"?

feel-dumb-just-being-in-the-same-newsgroup-ly y'rs - steve

--

http://www.holdenweb.com

http://pydish.holdenweb.com

Holden Web LLC +1 800 494 3119

[Martin v. Löwis]

Re: unicode question

Bengt Richter wrote:[color=blue][color=green]
>>Because print invokes str() on its argument, unless the argument is
>>already a byte string (in which case it prints it directly), or a[/color]
>
> ^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^ ^^^^^^-- effectively an assumption that
> bytestring.deco de('some_unknow n_encoding').en code(sys.stdout .encoding)
> has already been done, it seems (I'm not arguing against).[/color]

Not really. sys.stdout really is a byte string, which may or may
not *have* an encoding. Python tries to guess, and refuses to
in the face of ambiguity: e.g. if sys.stdout is a file, resulting
from

python mkimage.py > image.gif

then sys.stdout really does not *have* an encoding - but it still
is a byte stream. So copying the bytes to stdout is a
straight-forward thing to do.

Of course, "print" should only be used if the stream is meant to
transmit characters, and then the bytes written to the stream should
use the stream's encoding. This is indeed the assumption - but one
that the application author needs to make.
[color=blue]
> So how about changing print so that it doesn't blindly use str(y)[/color]

On the C level, this is already possible, through tp_print. Whether or
not this should be exposed to the Python level (or whether doing so
would just add to the confusion), I don't know.
[color=blue]
> but instead
> first tries to get y.__str__() in case the latter returns unicode?
> Then print y can succeed the way print y.__str__() does now.[/color]

As yet another alternative, print could invoke unicode(), if
there is a stream encoding. This would try __unicode__firs t,
then fall back to call __str__. Patches in this direction would
be welcome - but the code implementing print is already quite
involved, so a redesign (with a PEP and everything) might also
be in order.

In P3k, this part of the issue will go away, as str() then will
return Unicode strings.
[color=blue]
> I.e., str doesn't know that printing and '%s' can use unicode to good effect
> if it available, so for print and str.__mod__ blindly to use str() intermediately
> throws away an opportunity to do better ISTM.[/color]

That is true. Of course, there is already so much backwards
compatibility in this that any change to behaviour (such as
trying unicode() before trying str()) might break things.

Regards,
Martin

[Martin v. Löwis]

Re: unicode question

Steve Holden wrote:[color=blue]
> Am I the only person who found it scary that Bengt could apparently
> casually drop on a polynomial the would decode to " Löwis"?[/color]

I'm not scared, but honored, of course.

Regards,
Martin

[Bengt Richter]

Re: unicode question

On Tue, 23 Nov 2004 20:37:04 +0100, =?ISO-8859-1?Q?=22Martin_v =2E_L=F6wis=22? = <martin@v.loewi s.de> wrote:
[color=blue]
>Steve Holden wrote:[color=green]
>> Am I the only person who found it scary that Bengt could apparently
>> casually drop on a polynomial the would decode to " Löwis"?[/color][/color]
Well, don't give me too much credit, though I admit enjoying a little unearned
flattered-ego buzz ;-) But it's not a big deal if you had recently implemented
an automatic lambda-printer-outer to solve for a polynomial function f such that
f(0)==k0, f(1)==k1, .. f(n)==kn. For a single number k0 that will be lambda x: k0
and for two numbers k0, k1 will be lambda x: k0 + x*(k1-k0) etc. It's a matter of
solving some simultaneous equations for the coefficient values, which I had done
in response to a previous thread. For that, I happened to have had some experience
from the '60s writing variations on an equation solver (back when we congratulated
ourselves on getting all (software-implemented) floating point ops other than divide
to execute in under a millisecond ;-) Here I was using an exact decimal module I happened
to have (also built in response to previous thread discussion ;-), so I didn't even have
to look for maximum abs pivot elements in the matrix for this one. And it didn't have to be fast.
So it was kind of a fun exercise. But anyway, it was all ready to go at this point, so
all I had to was do was run coeffsx.py with the character ord values as args on the command line.
The opportunity to use it in a fun way to fake casual wizardry was just dumb luck ;-)
[color=blue]
>
>I'm not scared, but honored, of course.
>[/color]
A bit late responding, but I couldn't think of a clever followup to that ;-)
But Just to play fair,

print ''.join([chr((lambda x: (
-6244372133*x**3 1 +3013910052086* x**30 -695396351572920 *x**29
+10210575230774 1620*x**28 -107153038049746 59632*x**27 +85573431495191 9397204*x**26
-540677133391161 01354860*x**25 +27741212965686 07137441900*x** 24
-117725625258165 396333623970*x* *23 +41874052706021 60539007125440* x**22
-126060225187601 954901807327900 *x**21 +32349087369102 954690781831017 00*x**20
-711218789809664 181142050952976 40*x**19 +13442689029237 175711671172264 51980*x**18
-218866014040746 607512454037499 48900*x**17 +30718069894879 384184636891077 6059300*x**16
-371471921877217 015440606626937 1644945*x**15 +38641327091060 849304069885597 725238090*x**14
-344757809926306 996671359721670 334393500*x**13 +26270691157102 417044779211210 71756668600*x** 12
-169988694260954 318237542373700 45113150352*x** 11 +92697362475995 606001274610327 169882407584*x* *10
-421837211162827 653880286870838 716820642880*x* *9 +15816950333566 572014347364942 81105646218880* x**8
-480581774888383 763661453080520 4695373091328*x **7 +11572394080794 032785251889126 742747327087616 *x**6
-214178209444190 130803745251345 00006003159040* x**5 +29141767437911 436346798089144 038222112768000 *x**4
-271860864288260 943461084314476 44781404160000* x**3 +15339943556592 952236643053124 047771402240000 *x**2
-388225373807829 537910251710026 6822041600000*x +23023948231698 183889631576064 0000000)
/274094621805930 760590852096000 0000
)(x)) for x in xrange(32)])

Not-ready-to-be-mythologized-though-plenty-flatterable-ly y'rs

Regards,
Bengt Richter