Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

**Alex Martelli** · Jul 18 '05, 07:41 PM

Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

Cameron Laird <python-url@phaseit.net > wrote:
...[color=blue]
> Yippee! The martellibot promises to explain Unicode for Pythoneers.
> http://groups-beta.google.com/group/...15a5a05c206712[/color]

Uh -- _did_ I? Eeep... I guess I did... mostly, I was pointing to
Holger Krekel's very nice recipe (not sure he posted it to the site as
well as submitting it for the printed edition, but, lobby _HIM_ about
that;-).

Alex

**holger krekel** · Jul 18 '05, 07:48 PM

Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

On Fri, Dec 31, 2004 at 19:18 +0100, Alex Martelli wrote:[color=blue]
> Cameron Laird <python-url@phaseit.net > wrote:
> ...[color=green]
> > Yippee! The martellibot promises to explain Unicode for Pythoneers.
> > http://groups-beta.google.com/group/...15a5a05c206712[/color]
>
> Uh -- _did_ I? Eeep... I guess I did... mostly, I was pointing to
> Holger Krekel's very nice recipe (not sure he posted it to the site as
> well as submitting it for the printed edition, but, lobby _HIM_ about
> that;-).[/color]

FWIW, i added the recipe back to the online cookbook. It's not perfectly
formatted but still useful, i hope.

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/361742

cheers,

holger

P.S: happy new year.

**michele.simionato@gmail.com** · Jul 18 '05, 07:48 PM

Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

Holger:
[color=blue]
> FWIW, i added the recipe back to the online cookbook. It's not[/color]
perfectly[color=blue]
> formatted but still useful, i hope.[/color]
[color=blue]
> http://aspn.activestate.com/ASPN/Coo.../Recipe/361742[/color]

Uhm... on my system I get:
[color=blue][color=green][color=darkred]
>>> german_ae = unicode('\xc3\x a4', 'utf8')
>>> print german_ae # dunno if it will appear right on Google groups[/color][/color][/color]
ä
[color=blue][color=green][color=darkred]
>>> german_ae.decod e('latin1')[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xe4' in
position 0: ordinal not in range(128)
?? What's wrong?

Michele Simionato

**Stephan Diehl** · Jul 18 '05, 07:48 PM

Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

On Tue, 04 Jan 2005 05:43:32 -0800, michele.simiona to wrote:
[color=blue]
> Holger:
>[color=green]
>> FWIW, i added the recipe back to the online cookbook. It's not[/color]
> perfectly[color=green]
>> formatted but still useful, i hope.[/color]
>[color=green]
>> http://aspn.activestate.com/ASPN/Coo.../Recipe/361742[/color]
>
> Uhm... on my system I get:
>[color=green][color=darkred]
>>>> german_ae = unicode('\xc3\x a4', 'utf8')
>>>> print german_ae # dunno if it will appear right on Google groups[/color][/color]
> ä
>[color=green][color=darkred]
>>>> german_ae.decod e('latin1')[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xe4' in
> position 0: ordinal not in range(128)
> ?? What's wrong?[/color]

I'd rather use german_ae.encod e('latin1')
^^^^^^

which returns '\xe4'.[color=blue]
>
> Michele Simionato[/color]

**michele.simionato@gmail.com** · Jul 18 '05, 07:48 PM

Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

Stephan:
[color=blue]
> I'd rather use german_ae.encod e('latin1')[/color]
^^^^^^[color=blue]
> which returns '\xe4'.[/color]

uhm ... then there is a misprint in the discussion of the recipe;
BTW what's the difference between .encode and .decode ?
(yes, I have been living in happy ASCII-land until now ... ;)
I should probably ask for an unicode primer, I have found the
one by Marc André Lemburg

Page not found!

http://www.reportlab.com/i18n/python_unicode_tutorial.html

and I am reading it right now.

Michele Simionato

**Aahz** · Jul 18 '05, 07:48 PM

Unicode universe (was Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30))

In article <1104849206.111 461.70500@c13g2 000cwb.googlegr oups.com>,
<michele.simion ato@gmail.com> wrote:[color=blue]
>
>BTW what's the difference between .encode and .decode ?
>(yes, I have been living in happy ASCII-land until now ... ;)[/color]

Here's the stark simple recipe: when you use Unicode, you *MUST* switch
to a Unicode-centric view of the universe. Therefore you encode *FROM*
Unicode and you decode *TO* Unicode. Period. It's similar to the way
floating point contaminates ints.
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

"19. A language that doesn't affect the way you think about programming,
is not worth knowing." --Alan Perlis

**Skip Montanaro** · Jul 18 '05, 07:48 PM

Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

michele> BTW what's the difference between .encode and .decode ?

I started to answer, then got confused when I read the docstrings for
unicode.encode and unicode.decode:
[color=blue][color=green][color=darkred]
>>> help(u"\xe4".de code)[/color][/color][/color]
Help on built-in function decode:

decode(...)
S.decode([encoding[,errors]]) -> string or unicode

Decodes S using the codec registered for encoding. encoding defaults
to the default encoding. errors may be given to set a different error
handling scheme. Default is 'strict' meaning that encoding errors raise
a UnicodeDecodeEr ror. Other possible values are 'ignore' and 'replace'
as well as any other name registerd with codecs.register _error that is
able to handle UnicodeDecodeEr rors.
[color=blue][color=green][color=darkred]
>>> help(u"\xe4".en code)[/color][/color][/color]
Help on built-in function encode:

encode(...)
S.encode([encoding[,errors]]) -> string or unicode

Encodes S using the codec registered for encoding. encoding defaults
to the default encoding. errors may be given to set a different error
handling scheme. Default is 'strict' meaning that encoding errors raise
a UnicodeEncodeEr ror. Other possible values are 'ignore', 'replace' and
'xmlcharrefrepl ace' as well as any other name registered with
codecs.register _error that can handle UnicodeEncodeEr rors.

It probably makes sense to one who knows, but for the feeble-minded like
myself, they seem about the same.

I'd be happy to add a couple examples to the string methods section of the
docs if someone will produce something simple that makes the distinction
clear.

Skip

**michele.simionato@gmail.com** · Jul 18 '05, 07:48 PM

Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

Yep, I did the same and got confused :-/

Michele

**Skip Montanaro** · Jul 18 '05, 07:48 PM

Re: Unicode universe (was Re: Dr. Dobb's Python-URL! - weekly Pythonnews and links (Dec 30))

aahz> Here's the stark simple recipe: when you use Unicode, you *MUST*
aahz> switch to a Unicode-centric view of the universe. Therefore you
aahz> encode *FROM* Unicode and you decode *TO* Unicode. Period. It's
aahz> similar to the way floating point contaminates ints.

That's what I do in my code. Why do Unicode objects have a decode method
then?

Skip

**Thomas Heller** · Jul 18 '05, 07:48 PM

Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

Skip Montanaro <skip@pobox.com > writes:
[color=blue]
> michele> BTW what's the difference between .encode and .decode ?
>
> I started to answer, then got confused when I read the docstrings for
> unicode.encode and unicode.decode:
>[color=green][color=darkred]
> >>> help(u"\xe4".de code)[/color][/color]
> Help on built-in function decode:
>
> decode(...)
> S.decode([encoding[,errors]]) -> string or unicode
>
> Decodes S using the codec registered for encoding. encoding defaults
> to the default encoding. errors may be given to set a different error
> handling scheme. Default is 'strict' meaning that encoding errors raise
> a UnicodeDecodeEr ror. Other possible values are 'ignore' and 'replace'
> as well as any other name registerd with codecs.register _error that is
> able to handle UnicodeDecodeEr rors.
>[color=green][color=darkred]
> >>> help(u"\xe4".en code)[/color][/color]
> Help on built-in function encode:
>
> encode(...)
> S.encode([encoding[,errors]]) -> string or unicode
>
> Encodes S using the codec registered for encoding. encoding defaults
> to the default encoding. errors may be given to set a different error
> handling scheme. Default is 'strict' meaning that encoding errors raise
> a UnicodeEncodeEr ror. Other possible values are 'ignore', 'replace' and
> 'xmlcharrefrepl ace' as well as any other name registered with
> codecs.register _error that can handle UnicodeEncodeEr rors.
>
> It probably makes sense to one who knows, but for the feeble-minded like
> myself, they seem about the same.[/color]

It seems also the error messages aren't too helpful:
[color=blue][color=green][color=darkred]
>>> "ä".encode("lat in-1")[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x84 in position 0: ordinal not in range(128)[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

Hm, why does the 'encode' call complain about decoding?

Why do string objects have an encode method, and why do unicode objects
have a decode method, and what does this error message want to tell me:
[color=blue][color=green][color=darkred]
>>> u"ä".decode("la tin-1")[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode character u'\xe4' in position 0: ordinal not in range(128)[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

Thomas

**Max M** · Jul 18 '05, 07:48 PM

Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

michele.simiona to@gmail.com wrote:
[color=blue]
> uhm ... then there is a misprint in the discussion of the recipe;
> BTW what's the difference between .encode and .decode ?
> (yes, I have been living in happy ASCII-land until now ... ;)[/color]

# -*- coding: latin-1 -*-

# here i make a unicode string
unicode_file = u'Some danish characters æøå' #.encode('hex')
print type(unicode_fi le)
print repr(unicode_fi le)
print ''

# I can convert this unicode string to an ordinary string.
# because æøå are in the latin-1 charmap it can be understood as
# a latin-1 string
# the æøå characters even has the same value in both
latin1_file = unicode_file.en code('latin-1')
print type(latin1_fil e)
print repr(latin1_fil e)
print latin1_file
print ''

## I can *not* convert it to ascii
#ascii_file = unicode_file.en code('ascii')
#print ''

# I can also convert it to utf-8
utf8_file = unicode_file.en code('utf-8')
print type(utf8_file)
print repr(utf8_file)
print utf8_file
print ''

#utf8_file is now an ordinary string. again it can help to think of it
as a file
#format.
#
#I can convert this file/string back to unicode again by using the
decode method.
#It tells python to decode this "file format" as utf-8 when it loads it
onto a
#unicode string. And we are back where we started

unicode_file = utf8_file.decod e('utf-8')
print type(unicode_fi le)
print repr(unicode_fi le)
print ''

# So basically you can encode a unicode string into a special
string/file format
# and you can decode a string from a special string/file format back
into unicode.

############### ############### #####

<type 'unicode'>
u'Some danish characters \xe6\xf8\xe5'

<type 'str'>
'Some danish characters \xe6\xf8\xe5'
Some danish characters æøå

<type 'str'>
'Some danish characters \xc3\xa6\xc3\xb 8\xc3\xa5'
Some danish characters Ã¦Ã¸Ã¥

<type 'unicode'>
u'Some danish characters \xe6\xf8\xe5'

--

hilsen/regards Max M, Denmark

mxm.dk

http://www.mxm.dk/

mxm.dk

IT's Mad Science

**Max M** · Jul 18 '05, 07:49 PM

Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

Thomas Heller wrote:
[color=blue]
> It seems also the error messages aren't too helpful:
>[color=green][color=darkred]
>>>>"ä".encode( "latin-1")[/color][/color]
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x84 in position 0: ordinal not in range(128)
>
> Hm, why does the 'encode' call complain about decoding?[/color]

Because it tries to print it out to your console and fail. While writing
to the console it tries to convert to ascii.

Beside, you should write:

u"ä".encode("la tin-1") to get a latin-1 encoded string.

--

hilsen/regards Max M, Denmark

mxm.dk

http://www.mxm.dk/

mxm.dk

IT's Mad Science

**Thomas Heller** · Jul 18 '05, 07:49 PM

Re: Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

Max M <maxm@mxm.dk> writes:
[color=blue]
> Thomas Heller wrote:
>[color=green]
>> It seems also the error messages aren't too helpful:
>>[color=darkred]
>>>>>"ä".encode ("latin-1")[/color]
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in ?
>> UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x84 in position 0: ordinal not in range(128)
>> Hm, why does the 'encode' call complain about decoding?[/color]
>
> Because it tries to print it out to your console and fail. While
> writing to the console it tries to convert to ascii.[/color]

Wrong, same error without trying to print something:
[color=blue][color=green][color=darkred]
>>> x = "ä".encode("lat in-1")[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0x84 in position 0: ordinal not in range(128)[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]
[color=blue]
>
> Beside, you should write:
>
> u"ä".encode("la tin-1") to get a latin-1 encoded string.[/color]

I know, but the question was: why does a unicode string has a encode
method, and why does it complain about decoding (which has already been
answered in the meantime).

Thomas

**Walter Dörwald** · Jul 18 '05, 07:49 PM

Re: Unicode universe (was Re: Dr. Dobb's Python-URL! - weekly Pythonnews and links (Dec 30))

Skip Montanaro wrote:[color=blue]
> aahz> Here's the stark simple recipe: when you use Unicode, you *MUST*
> aahz> switch to a Unicode-centric view of the universe. Therefore you
> aahz> encode *FROM* Unicode and you decode *TO* Unicode. Period. It's
> aahz> similar to the way floating point contaminates ints.
>
> That's what I do in my code. Why do Unicode objects have a decode method
> then?[/color]

Because MAL implemented it! >;->

It first encodes in the default encoding and then decodes the result
with the specified encoding, so if u is a unicode object
u.decode("utf-16")
is an abbreviation of
u.encode().deco de("utf-16")

In the same way str has an encode method, so
s.encode("utf-16")
is an abbreviation of
s.decode().enco de("utf-16")

Bye,
Walter Dörwald

Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

Dr. Dobb's Python-URL! - weekly Python news and links (Dec 30)

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment