Unicode problem with exec

**Diez B. Roggisch** · Jun 23 '06, 01:05 PM

Re: Unicode problem with exec

Thomas Heller schrieb:[color=blue]
> I'm using code.Interactiv e console but it doesn't work correctly
> with non-ascii characters. I think it boils down to this problem:
>
> Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on
> win32
> Type "help", "copyright" , "credits" or "license" for more information.[color=green][color=darkred]
>>>> print u"ä"[/color][/color]
> ä[color=green][color=darkred]
>>>> exec 'print u"ä"'[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "<string>", line 1, in ?
> File "c:\python24\li b\encodings\cp8 50.py", line 18, in encode
> return codecs.charmap_ encode(input,er rors,encoding_m ap)
> UnicodeEncodeEr ror: 'charmap' codec can't encode character u'\x84' in
> position 0: character maps to <undefined>[color=green][color=darkred]
>>>> ^Z[/color][/color]
>
> Why does the exec call fail, and is there a workaround?[/color]

Most probably because you failed to encode the snippet as whole - so the
embedded unicode literal isn't encoded properly.

As your exec-encoding seems to be cp850, maybe

exec u"print u'ä'".encode("c p850")

works.

Diez

**John Machin** · Jun 23 '06, 10:55 PM

Re: Unicode problem with exec

On 23/06/2006 9:06 PM, Thomas Heller wrote:[color=blue]
> I'm using code.Interactiv e console but it doesn't work correctly
> with non-ascii characters. I think it boils down to this problem:
>
> Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on
> win32
> Type "help", "copyright" , "credits" or "license" for more information.[color=green][color=darkred]
>>>> print u"ä"[/color][/color][/color]

This is utterly useless for diagnostic purposes. What you see is NOT
what you've got. Use repr().

What you've got, as the error message says, is u'\x84' which is not
u"\N{LATIN SMALL LETTER A WITH DIAERESIS}", it is a control character.

See below.
[color=blue]
> ä[color=green][color=darkred]
>>>> exec 'print u"ä"'[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "<string>", line 1, in ?
> File "c:\python24\li b\encodings\cp8 50.py", line 18, in encode
> return codecs.charmap_ encode(input,er rors,encoding_m ap)
> UnicodeEncodeEr ror: 'charmap' codec can't encode character u'\x84' in
> position 0: character maps to <undefined>[color=green][color=darkred]
>>>> ^Z[/color][/color]
>
> Why does the exec call fail, and is there a workaround?
>[/color]

Executive summary:

The exec statement didn't fail, it was the print statement trying to
print, to your CP850 console, a unicode char that doesn't exist in CP850.

This happened because you copied a character whose repr() is '\x84' from
your MS-DOS console and pasted it into 'u"<insert any old rubbish
here>"' :-)

Details:

Windows XP, in a console screen:

Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright" , "credits" or "license" for more information.
|>> uc = u"\N{LATIN SMALL LETTER A WITH DIAERESIS}"
|>> uc
u'\xe4' <<== agrees with Unicode book
|>> encoded = uc.encode('cp85 0')[color=blue][color=green][color=darkred]
>>> encoded[/color][/color][/color]
'\x84' <<== agrees with

http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP850.TXT

|>> print uc
ä <<== looks like LATIN SMALL LETTER A WITH DIAERESIS, as expected
|>> print encoded
ä <<== looks like LATIN SMALL LETTER A WITH DIAERESIS, as expected
|>> print u"\x84"
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "c:\python24\li b\encodings\cp8 50.py", line 18, in encode
return codecs.charmap_ encode(input,er rors,encoding_m ap)
UnicodeEncodeEr ror: 'charmap' codec can't encode character u'\x84' in
position 0: character maps to <undefined>
<<== as expected

Looks like Python is working fine to me ...
So, what's happening? Look at this:

|>> char1 = u"ä" <<= corresponds to your "print"
|>> char2 = "ä" <<= corresponds to your exec -- which was given a STRING
constant, like this, not a Unicode constant.

Character in char1 was copied from DOS console.
Second line was obtained by DOS console editing of copy of first line.

|>> char1
u'\xe4'
|>> char2
'\x84' <<= Aha!

What you have done is effectively: exec 'print u"\x84"'

Workaround/kludge/bypass:

exec u'print u"ä"'
......^

Much better: embed non-ASCII characters in source code *ONLY* when you
have a proper coding header: http://www.python.org/dev/peps/pep-0263/

HTH,
John

**John Machin** · Jun 23 '06, 10:55 PM

Re: Unicode problem with exec

On 23/06/2006 9:06 PM, Thomas Heller wrote:[color=blue]
> I'm using code.Interactiv e console but it doesn't work correctly
> with non-ascii characters. I think it boils down to this problem:
>
> Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on
> win32
> Type "help", "copyright" , "credits" or "license" for more information.[color=green][color=darkred]
>>>> print u"ä"[/color][/color][/color]

This is utterly useless for diagnostic purposes. What you see is NOT
what you've got. Use repr().

What you've got, as the error message says, is u'\x84' which is not
u"\N{LATIN SMALL LETTER A WITH DIAERESIS}", it is a control character.

See below.
[color=blue]
> ä[color=green][color=darkred]
>>>> exec 'print u"ä"'[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "<string>", line 1, in ?
> File "c:\python24\li b\encodings\cp8 50.py", line 18, in encode
> return codecs.charmap_ encode(input,er rors,encoding_m ap)
> UnicodeEncodeEr ror: 'charmap' codec can't encode character u'\x84' in
> position 0: character maps to <undefined>[color=green][color=darkred]
>>>> ^Z[/color][/color]
>
> Why does the exec call fail, and is there a workaround?
>[/color]

Executive summary:

The exec statement didn't fail, it was the print statement trying to
print, to your CP850 console, a unicode char that doesn't exist in CP850.

This happened because you copied a character whose repr() is '\x84' from
your MS-DOS console and pasted it into 'u"<insert any old rubbish
here>"' :-)

Details:

Windows XP, in a console screen:

Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright" , "credits" or "license" for more information.
|>> uc = u"\N{LATIN SMALL LETTER A WITH DIAERESIS}"
|>> uc
u'\xe4' <<== agrees with Unicode book
|>> encoded = uc.encode('cp85 0')[color=blue][color=green][color=darkred]
>>> encoded[/color][/color][/color]
'\x84' <<== agrees with

http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP850.TXT

|>> print uc
ä <<== looks like LATIN SMALL LETTER A WITH DIAERESIS, as expected
|>> print encoded
ä <<== looks like LATIN SMALL LETTER A WITH DIAERESIS, as expected
|>> print u"\x84"
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "c:\python24\li b\encodings\cp8 50.py", line 18, in encode
return codecs.charmap_ encode(input,er rors,encoding_m ap)
UnicodeEncodeEr ror: 'charmap' codec can't encode character u'\x84' in
position 0: character maps to <undefined>
<<== as expected

Looks like Python is working fine to me ...
So, what's happening? Look at this:

|>> char1 = u"ä" <<= corresponds to your "print"
|>> char2 = "ä" <<= corresponds to your exec -- which was given a STRING
constant, like this, not a Unicode constant.

Character in char1 was copied from DOS console.
Second line was obtained by DOS console editing of copy of first line.

|>> char1
u'\xe4'
|>> char2
'\x84' <<= Aha!

What you have done is effectively: exec 'print u"\x84"'

Workaround/kludge/bypass:

exec u'print u"ä"'
......^

Much better: embed non-ASCII characters in source code *ONLY* when you
have a proper coding header: http://www.python.org/dev/peps/pep-0263/

HTH,
John

Unicode problem with exec

Unicode problem with exec

Comment

Comment

Comment