unicode encoding usablilty problem

Re: unicode encoding usablilty problem

anonymous coward <aurora00@gmail .com> wrote:
[color=blue]
> This brings up another issue. Most references and books focus exclusive on entering unicode
> literal and using the encode/decode methods. The fallacy is that string is such a basic data type
> use throughout the program, you really don't want to make a individual decision everytime when
> you use string (and take a penalty for any negligence). The Java has a much more usable model
> with unicode used internally and encoding/decoding decision only need twice when dealing with
> input and output.[/color]

that's how you should do things in Python too, of course. a unicode string
uses unicode internally. decode on the way in, encode on the way out, and
things just work.

the fact that you can mess things up by mixing unicode strings with binary
strings doesn't mean that you have to mix unicode strings with binary strings
in your program.
[color=blue]
> Even for those who choose to use unicode, it is almost impossible to ensure their program work
> correctly.[/color]

well, if you use unicode the way it was intended to, it just works.

</F>

Re: unicode encoding usablilty problem

On Fri, 18 Feb 2005 19:24:10 +0100, Fredrik Lundh <fredrik@python ware.com>
wrote:
[color=blue]
> that's how you should do things in Python too, of course. a unicode
> string
> uses unicode internally. decode on the way in, encode on the way out, and
> things just work.
>
> the fact that you can mess things up by mixing unicode strings with
> binary
> strings doesn't mean that you have to mix unicode strings with binary
> strings
> in your program.[/color]

I don't want to mix them. But how could I find them? How do I know this
statement can be potential problem

if a==b:

where a and b can be instantiated individually far away from this line of
code that put them together?

In Java they are distinct data type and the compiler would catch all
incorrect usage. In Python, the interpreter seems to 'help' us to promote
binary string to unicode. Things works fine, unit tests pass, all until
the first non-ASCII characters come in and then the program breaks.

Is there a scheme for Python developer to use so that they are safe from
incorrect mixing?

Re: unicode encoding usablilty problem

aurora wrote:
[color=blue]
> [...]
> In Java they are distinct data type and the compiler would catch all
> incorrect usage. In Python, the interpreter seems to 'help' us to
> promote binary string to unicode. Things works fine, unit tests pass,
> all until the first non-ASCII characters come in and then the program
> breaks.
>
> Is there a scheme for Python developer to use so that they are safe
> from incorrect mixing?[/color]

Put the following:

import sys
sys.setdefaulte ncoding("undefi ned")

in a file named sitecustomize.p y somewhere in your Python path and
Python will complain whenever there's an implicit conversion between
str and unicode.

HTH,
Walter Dörwald

Re: unicode encoding usablilty problem

Fredrik Lundh napisa³(a):
[color=blue][color=green]
>>This brings up another issue. Most references and books focus exclusive on entering unicode
>>literal and using the encode/decode methods. The fallacy is that string is such a basic data type
>>use throughout the program, you really don't want to make a individual decision everytime when
>>you use string (and take a penalty for any negligence). The Java has a much more usable model
>>with unicode used internally and encoding/decoding decision only need twice when dealing with
>>input and output.[/color]
>
> that's how you should do things in Python too, of course. a unicode string
> uses unicode internally. decode on the way in, encode on the way out, and
> things just work.[/color]

There are implementations of Python where it isn't so easy, Python for
iSeries (http://www.iseriespython.com/) is one of them. The code written
for "normal" platform doesn't work on AS/400, even if all strings used
internally are unicode objects, also unicode literals don't work as
expected.

Of course, this is implementation fault but this makes a headache if you
need to write portable code.

--
Jarek Zgoda
http://jpa.berlios.de/ | http://www.zgodowie.org/

Re: unicode encoding usablilty problem

=?ISO-8859-15?Q?Walter_D=F 6rwald?= <walter@livingl ogic.de> writes:
[color=blue]
> aurora wrote:
>[color=green]
> > [...]
>> In Java they are distinct data type and the compiler would catch all
>> incorrect usage. In Python, the interpreter seems to 'help' us to
>> promote binary string to unicode. Things works fine, unit tests
>> pass, all until the first non-ASCII characters come in and then the
>> program breaks.
>> Is there a scheme for Python developer to use so that they are safe
>> from incorrect mixing?[/color]
>
> Put the following:
>
> import sys
> sys.setdefaulte ncoding("undefi ned")
>
> in a file named sitecustomize.p y somewhere in your Python path and
> Python will complain whenever there's an implicit conversion between
> str and unicode.[/color]

Sounds cool, so I did it.
And started a program I was currently working on.
The first function in it is this:

if sys.platform == "win32":

def _locate_gccxml( ):
import _winreg
for subkey in (r"Software\gcc xml", r"Software\Kitw are\GCC_XML"):
for root in (_winreg.HKEY_C URRENT_USER, _winreg.HKEY_LO CAL_MACHINE):
try:
hkey = _winreg.OpenKey (root, subkey, 0, _winreg.KEY_REA D)
except WindowsError, detail:
if detail.errno != 2:
raise
else:
return _winreg.QueryVa lueEx(hkey, "loc")[0] + r"\bin"

loc = _locate_gccxml( )
if loc:
os.environ["PATH"] = loc

All strings in that snippet are text strings, so the first approach was
to convert them to unicode literals. Doesn't work. Here is the final,
working version (changes are marked):

if sys.platform == "win32":

def _locate_gccxml( ):
import _winreg
for subkey in (r"Software\gcc xml", r"Software\Kitw are\GCC_XML"):
for root in (_winreg.HKEY_C URRENT_USER, _winreg.HKEY_LO CAL_MACHINE):
try:
hkey = _winreg.OpenKey (root, subkey, 0, _winreg.KEY_REA D)
except WindowsError, detail:
if detail.errno != 2:
raise
else:
return _winreg.QueryVa lueEx(hkey, "loc")[0] + ur"\bin"
#-----------------------------------------------------------------^
loc = _locate_gccxml( )
if loc:
os.environ["PATH"] = loc.encode("mbc s")
#--------------------------------^

So, it appears that:

- the _winreg.QueryVa lue function is strange: it takes ascii strings,
but returns a unicode string.
- _winreg.OpenKey takes ascii strings
- the os.environ["PATH"] accepts an ascii string.

And I won't guess what happens when there are registry entries with
unlauts (ok, they could be handled by 'mbcs' encoding), and with chinese
or japanese characters (no way to represent them in ascii strings with a
western locale and mbcs encoding, afaik).


I suggest that 'sys.setdefault encoding("undef ined")' be the standard
setting for the core developers ;-)

Thomas

Re: unicode encoding usablilty problem

aurora wrote:[color=blue]
> The Java
> has a much more usable model with unicode used internally and
> encoding/decoding decision only need twice when dealing with input and
> output.[/color]

In addition to Fredrik's comment (that you should use the same model
in Python) and Walter's comment (that you can enforce it by setting
the default encoding to "undefined" ), I'd like to point out the
historical reason: Python predates Unicode, so the byte string type
has many convenience operations that you would only expect of
a character string.

We have come up with a transition strategy, allowing existing
libraries to widen their support from byte strings to character
strings. This isn't a simple task, so many libraries still expect
and return byte strings, when they should process character strings.
Instead of breaking the libraries right away, we have defined
a transitional mechanism, which allows to add Unicode support
to libraries as the need arises. This transition is still in
progress.

Eventually, the primary string type should be the Unicode
string. If you are curious how far we are still off that goal,
just try running your program with the -U option.

Regards,
Martin

Re: unicode encoding usablilty problem

Walter Dörwald napisa³(a):
[color=blue][color=green]
>> Is there a scheme for Python developer to use so that they are safe
>> from incorrect mixing?[/color]
>
>
> Put the following:
>
> import sys
> sys.setdefaulte ncoding("undefi ned")
>
> in a file named sitecustomize.p y somewhere in your Python path and
> Python will complain whenever there's an implicit conversion between
> str and unicode.[/color]

This will help in your code, but there is big pile of modules in stdlib
that are not unicode-friendly. From my daily practice come shlex
(tokenizer works only with encoded strings) and logging (you cann't
specify encoding for FileHandler).

--
Jarek Zgoda
http://jpa.berlios.de/ | http://www.zgodowie.org/

Re: unicode encoding usablilty problem

=?ISO-8859-15?Q?=22Martin_ v=2E_L=F6wis=22 ?= <martin@v.loewi s.de> writes:
[color=blue]
> We have come up with a transition strategy, allowing existing
> libraries to widen their support from byte strings to character
> strings. This isn't a simple task, so many libraries still expect
> and return byte strings, when they should process character strings.
> Instead of breaking the libraries right away, we have defined
> a transitional mechanism, which allows to add Unicode support
> to libraries as the need arises. This transition is still in
> progress.
>
> Eventually, the primary string type should be the Unicode
> string. If you are curious how far we are still off that goal,
> just try running your program with the -U option.[/color]

Is it possible to specify a byte string literal when running with the -U option?

Thomas

Re: unicode encoding usablilty problem

=?ISO-8859-15?Q?=22Martin_ v=2E_L=F6wis=22 ?= <martin@v.loewi s.de> writes:
[color=blue]
> Eventually, the primary string type should be the Unicode
> string. If you are curious how far we are still off that goal,
> just try running your program with the -U option.[/color]

Not very far - can't even call functions ;-)

c:\>py -U
Python 2.5a0 (#60, Dec 29 2004, 11:27:13) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> def f(**kw):[/color][/color][/color]
.... pass
....[color=blue][color=green][color=darkred]
>>> f(**{"a": 0})[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: f() keywords must be strings[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

Thomas

Re: unicode encoding usablilty problem

Martin v. Löwis:
[color=blue]
> Eventually, the primary string type should be the Unicode
> string. If you are curious how far we are still off that goal,
> just try running your program with the -U option.[/color]

Tried both -U and sys.setdefaulte ncoding("undefi ned") on a couple of my
most used programs and saw a few library problems. One program reads job
advertisements from a mailing list, ranks them according to keywords, and
displays them using unicode to ensure that HTML entities like &bull; are
displayed correctly. That program worked without changes.

The second program reads my spam filled mail box removing messages that
match a set of header criteria. It uses decode_header and make_header from
the email.Header library module to convert each header from a set of encoded
strings into a single unicode string. As email.Header is strongly concerned
with unicode, I expected it would be able to handle the two modifications
well.

With -U, there was one bug in my code assuming that a string would be 8
bit and that was easily fixed. In email.Charset, __init__ expects a
non-unicode argument as it immediately calls unicode(input_c harset, 'ascii')
which fails when the argument is unicode. This can be fixed explicitly in
the __init__ but I would argue for a more lenient approach with unicode(u,
enc, err) always ignoring the enc and err arguments when the input is
already in unicode. Next sre breaks when building a mapping array because
array.array can not have a unicode type code. This should probably be fixed
in array rather than sre as mapping = array.array('b' .encode('ascii' ),
mapping).tostri ng() is too ugly. The final issue was in encodings.idna where
there is ace_prefix = "xn--"; uace_prefix = unicode(ace_pre fix, "ascii")
which again could avoid breakage if unicode was more lenient.

With sys.setdefaulte ncoding("undefi ned"), there were more problems and
they were harder to work around. One addition that could help would be a
function similar to str but with an optional encoding that would be used
when the input failed to convert to string because of a UnicodeError.
Something like

def stri(x, enc='us-ascii'):
try:
return str(x)
except UnicodeError:
return unicode(x).enco de(enc)

Neil

Re: unicode encoding usablilty problem

On Fri, 18 Feb 2005 20:18:28 +0100, Walter Dörwald <walter@livingl ogic.de>
wrote:
[color=blue]
> aurora wrote:
>[color=green]
> > [...]
>> In Java they are distinct data type and the compiler would catch all
>> incorrect usage. In Python, the interpreter seems to 'help' us to
>> promote binary string to unicode. Things works fine, unit tests pass,
>> all until the first non-ASCII characters come in and then the program
>> breaks.
>> Is there a scheme for Python developer to use so that they are safe
>> from incorrect mixing?[/color]
>
> Put the following:
>
> import sys
> sys.setdefaulte ncoding("undefi ned")
>
> in a file named sitecustomize.p y somewhere in your Python path and
> Python will complain whenever there's an implicit conversion between
> str and unicode.
>
> HTH,
> Walter Dörwald[/color]

That helps! Running unit test caught quite a few potential problems (as
well as a lot of safe of ASCII string promotion).

Re: unicode encoding usablilty problem

On Fri, 18 Feb 2005 21:16:01 +0100, Martin v. Löwis <martin@v.loewi s.de>
wrote:
[color=blue]
> I'd like to point out the
> historical reason: Python predates Unicode, so the byte string type
> has many convenience operations that you would only expect of
> a character string.
>
> We have come up with a transition strategy, allowing existing
> libraries to widen their support from byte strings to character
> strings. This isn't a simple task, so many libraries still expect
> and return byte strings, when they should process character strings.
> Instead of breaking the libraries right away, we have defined
> a transitional mechanism, which allows to add Unicode support
> to libraries as the need arises. This transition is still in
> progress.[/color]

I understand. So I wasn't yelling "why can't Python be more like Java". On
the other hand I also want to point out making individual decision for
each string wasn't practical and is very error prone. The fact that
unicode and 8 bit string look alike and work alike in common situation but
only run into problem with non-ASCII is very confusing for most people.

[color=blue]
> Eventually, the primary string type should be the Unicode
> string. If you are curious how far we are still off that goal,
> just try running your program with the -U option.[/color]

Lots of errors. Amount them are gzip (binary?!) and strftime??

I actually quite appriciate Python's power in processing binary data as
8-bit strings. But perhaps we should transition to use unicode as text
string as treat binary string as exception. Right now we have

'' - 8bit string; u'' unicode string

How about

b'' - 8bit string; '' unicode string

and no automatic conversion. Perhaps this can be activated by something
like the encoding declarations, so that transition can happen module by
module.

[color=blue]
> Regards,
> Martin[/color]

Re: unicode encoding usablilty problem

On Fri, 18 Feb 2005 21:43:52 +0100, Thomas Heller wrote:
[color=blue][color=green]
>> Eventually, the primary string type should be the Unicode
>> string. If you are curious how far we are still off that goal,
>> just try running your program with the -U option.[/color]
>
> Not very far - can't even call functions ;-)
>[color=green][color=darkred]
>>>> def f(**kw):[/color][/color]
> ... pass
> ...[color=green][color=darkred]
>>>> f(**{"a": 0})[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> TypeError: f() keywords must be strings[/color]

That is possible:[color=blue][color=green][color=darkred]
>>> f(**{chr(ord("a ")): 0})
>>>[/color][/color][/color]

WFM. SCNR,
Alexander

Re: unicode encoding usablilty problem

"aurora" <aurora00@gmail .com> wrote:
[color=blue]
> I don't want to mix them. But how could I find them? How do I know this statement can be
> potential problem
>
> if a==b:
>
> where a and b can be instantiated individually far away from this line of code that put them
> together?[/color]

if you don't know what a and b comes from, how can you be sure that
your program works at all? how can you be sure they're both strings?

("a op b" can fail in many ways, depending on what "a", "b", and "op" are)
[color=blue]
> Things works fine, unit tests pass, all until the first non-ASCII characters
> come in and then the program breaks.[/color]

if you have unit tests, why don't they include Unicode tests?

</F>