unicode encoding usablilty problem

[Nick Coghlan]

Re: unicode encoding usablilty problem

Thomas Heller wrote:[color=blue]
> =?ISO-8859-15?Q?=22Martin_ v=2E_L=F6wis=22 ?= <martin@v.loewi s.de> writes:
>
>[color=green]
>>We have come up with a transition strategy, allowing existing
>>libraries to widen their support from byte strings to character
>>strings. This isn't a simple task, so many libraries still expect
>>and return byte strings, when they should process character strings.
>>Instead of breaking the libraries right away, we have defined
>>a transitional mechanism, which allows to add Unicode support
>>to libraries as the need arises. This transition is still in
>>progress.
>>
>>Eventually, the primary string type should be the Unicode
>>string. If you are curious how far we are still off that goal,
>>just try running your program with the -U option.[/color]
>
>
> Is it possible to specify a byte string literal when running with the -U option?[/color]

Not that I know of. If the 'bytes' type happens, then I'd be a fan of b"" to get
a byte string instead of a character string.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan@email. com | Brisbane, Australia
---------------------------------------------------------------

http://boredomandlaziness.skystorm.net

[Martin v. Löwis]

Re: unicode encoding usablilty problem

Thomas Heller wrote:[color=blue]
> Is it possible to specify a byte string literal when running with the -U option?[/color]

Not literally. However, you can specify things like

bytes = [0x47, 0x49, 0x4f, 0x50, 0x01, 0x00]
bytes = ''.join((chr(x) for x in bytes))

Alternatively, you could rely on the 1:1 feature of Latin-1:

bytes = "GIOP\x01\0".en code("l1")

Regards,
Martin

[Martin v. Löwis]

Re: unicode encoding usablilty problem

aurora wrote:[color=blue]
> Lots of errors. Amount them are gzip (binary?!) and strftime??[/color]

For gzip, this is not surprising. It contains things like

self.fileobj.wr ite('\037\213')

which is not intended to denote characters.

[color=blue]
> How about
>
> b'' - 8bit string; '' unicode string
>
> and no automatic conversion.[/color]

This has been proposed before, see PEP 332. The problem is that
people often want byte strings to be mutable as well, so it is
still unclear whether it is better to make the b prefix denote
the current string type (so it would be currently redundant)
or a newly-created mutable string type (similar to array.array).
[color=blue]
> Perhaps this can be activated by something
> like the encoding declarations, so that transition can happen module by
> module.[/color]

That could work for the literals - a __future__ import would be
most appropriate. For "no automatic conversion", this is very
difficult to implement on a per-module basis. The errors typically
don't occur in the module itself, but in some function called by
the module (e.g. a builtin method of the string type). So the
callee would have to know whether the caller has a future
import...

Regards,
Martin

[Nick Coghlan]

Re: unicode encoding usablilty problem

Martin v. Löwis wrote:[color=blue][color=green]
>> How about
>>
>> b'' - 8bit string; '' unicode string
>>
>> and no automatic conversion.[/color]
>
>
> This has been proposed before, see PEP 332. The problem is that
> people often want byte strings to be mutable as well, so it is
> still unclear whether it is better to make the b prefix denote
> the current string type (so it would be currently redundant)
> or a newly-created mutable string type (similar to array.array).[/color]

Having "", u"", and r"" be immutable, while b"" was mutable would seem rather
inconsistent.

If you want a phased migration to 'assert (str is unicode) == True', then PEP
332 seems to have that covered:

1. Introduce 'bytes' as an alias of str
2. Introduce b"" as an alternate spelling of r""
3. Switch str to be an alias of unicode
4. Switch "" to be an alternate spelling of u""

Trying to intermingle this with making the bytes type mutable seems to be
begging for trouble - consider how many string-keyed dictionaries would break
with that change (the upgrade path is non-existent - you can't stay with str,
because you want byte strings, but you can't go to bytes, because you need
something immutable).

An alternative would be to have "bytestr" be the immutable type corresponding to
the current str (with b"" literals producing bytestr's), while reserving the
"bytes" name for a mutable byte sequence. That is, change PEP 332's upgrade path
to look more like:

* Add a bytestr builtin which is just a synonym for str. (2.5)
* Add a b"..." string literal which is equivalent to raw string literals,
with the exception that values which conflict with the source encoding of the
containing file not generate warnings. (2.5)
* Warn about the use of variables named "bytestr". (2.5 or 2.6)
* Introduce a bytestr builtin which refers to a sequence distinct from the
str type. (2.6)
* Make str a synonym for unicode. (3.0)

And separately:
* Introduce a bytes builtin which is a mutable byte sequence

Alternately, add array.bytes as a subclass of array.array, that provides a nicer
API for dealing specifically with byte strings.

The main point being, the replacement for 'str' needs to be immutable or the
upgrade process is going to be a serious PITA.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan@email. com | Brisbane, Australia
---------------------------------------------------------------

http://boredomandlaziness.skystorm.net

[Martin v. Löwis]

Re: unicode encoding usablilty problem

Nick Coghlan wrote:[color=blue]
> Having "", u"", and r"" be immutable, while b"" was mutable would seem
> rather inconsistent.[/color]

Yes. However, this inconsistency might be desirable. It would, of
course, mean that the literal cannot be a singleton. Instead, it has
to be a display (?), similar to list or dict displays: each execution
of the byte string literal creates a new object.
[color=blue]
> An alternative would be to have "bytestr" be the immutable type
> corresponding to the current str (with b"" literals producing
> bytestr's), while reserving the "bytes" name for a mutable byte
> sequence.[/color]

Indeed. This maze of options has caused the process to get stuck.
People also argue that with such an approach, we could as well
tell users to use array.array for the mutable type. But then,
people complain that it doesn't have all the library support that
strings have.
[color=blue]
> The main point being, the replacement for 'str' needs to be immutable or
> the upgrade process is going to be a serious PITA.[/color]

Somebody really needs to take this in his hands, completing the PEP,
writing a patch, checking applications to find out what breaks.

Regards,
Martin

[Martin v. Löwis]

Re: unicode encoding usablilty problem

Nick Coghlan wrote:[color=blue]
> Having "", u"", and r"" be immutable, while b"" was mutable would seem
> rather inconsistent.[/color]

Yes. However, this inconsistency might be desirable. It would, of
course, mean that the literal cannot be a singleton. Instead, it has
to be a display (?), similar to list or dict displays: each execution
of the byte string literal creates a new object.
[color=blue]
> An alternative would be to have "bytestr" be the immutable type
> corresponding to the current str (with b"" literals producing
> bytestr's), while reserving the "bytes" name for a mutable byte
> sequence.[/color]

Indeed. This maze of options has caused the process to get stuck.
People also argue that with such an approach, we could as well
tell users to use array.array for the mutable type. But then,
people complain that it doesn't have all the library support that
strings have.
[color=blue]
> The main point being, the replacement for 'str' needs to be immutable or
> the upgrade process is going to be a serious PITA.[/color]

Somebody really needs to take this in his hands, completing the PEP,
writing a patch, checking applications to find out what breaks.

Regards,
Martin

[Nick Coghlan]

Re: unicode encoding usablilty problem

Martin v. Löwis wrote:[color=blue]
> People also argue that with such an approach, we could as well
> tell users to use array.array for the mutable type. But then,
> people complain that it doesn't have all the library support that
> strings have.[/color]

Indeed - I've got a data manipulating program that I figured I could make
slightly less memory hungry by using arrays instead of strings.

I discovered very quickly just how inconvenient such a change would be in terms
of the available API for manipulation of the byte array (the loss of 'join'
support was a serious drawback). The program still uses strings for that reason.

However, I wonder if that might not be better solved by providing an
"array.bytearra y" that supported relevant portions of the string API (and easy
conversion to a string), rather than blurring the concept of immutable strings.

Hmm - something else the PEP needs to discuss: What happens to __str__ and
__unicode__? Is there a new __bytes__ slot?

I wonder if Skip is still up for championing this one. . .

Cheers,
Nick.
One PEP's enough for me (even though 338 doesn't seem to generate much interest)

--
Nick Coghlan | ncoghlan@email. com | Brisbane, Australia
---------------------------------------------------------------

http://boredomandlaziness.skystorm.net

[aurora]

Re: unicode encoding usablilty problem

On Sat, 19 Feb 2005 18:44:27 +0100, Fredrik Lundh <fredrik@python ware.com>
wrote:
[color=blue]
> "aurora" <aurora00@gmail .com> wrote:
>[color=green]
>> I don't want to mix them. But how could I find them? How do I know
>> this statement can be
>> potential problem
>>
>> if a==b:
>>
>> where a and b can be instantiated individually far away from this line
>> of code that put them
>> together?[/color]
>
> if you don't know what a and b comes from, how can you be sure that
> your program works at all? how can you be sure they're both strings?
>
> ("a op b" can fail in many ways, depending on what "a", "b", and "op"
> are)
>[/color]

a and b are both string. The issue is 8-bit string or unicode string.

[color=blue][color=green]
>> Things works fine, unit tests pass, all until the first non-ASCII
>> characters
>> come in and then the program breaks.[/color]
>
> if you have unit tests, why don't they include Unicode tests?
>
> </F>[/color]

How do I structure the test cases to guarantee coverage? It is not
practical to test every combinations of unicode/8-bit strings. Adding
non-ascii characters to test data probably make problem pop up earlier.
But it is arduous and it is hard to spot if you left out any.

[aurora]

Re: unicode encoding usablilty problem

On Sun, 20 Feb 2005 15:01:09 +0100, Martin v. Löwis <martin@v.loewi s.de>
wrote:
[color=blue]
> Nick Coghlan wrote:[color=green]
>> Having "", u"", and r"" be immutable, while b"" was mutable would seem
>> rather inconsistent.[/color]
>
> Yes. However, this inconsistency might be desirable. It would, of
> course, mean that the literal cannot be a singleton. Instead, it has
> to be a display (?), similar to list or dict displays: each execution
> of the byte string literal creates a new object.
>[color=green]
>> An alternative would be to have "bytestr" be the immutable type
>> corresponding to the current str (with b"" literals producing
>> bytestr's), while reserving the "bytes" name for a mutable byte
>> sequence.[/color]
>
> Indeed. This maze of options has caused the process to get stuck.
> People also argue that with such an approach, we could as well
> tell users to use array.array for the mutable type. But then,
> people complain that it doesn't have all the library support that
> strings have.
>[color=green]
>> The main point being, the replacement for 'str' needs to be immutable
>> or the upgrade process is going to be a serious PITA.[/color]
>
> Somebody really needs to take this in his hands, completing the PEP,
> writing a patch, checking applications to find out what breaks.
>
> Regards,
> Martin[/color]

What is the processing of getting a PEP work out? Does the work and
discussion carry out in the python-dev mailing list? I would be glad to
help out especially on this particular issue.

[Fredrik Lundh]

Re: unicode encoding usablilty problem

"aurora" <aurora00@gmail .com> wrote:
[color=blue][color=green]
>> if you don't know what a and b comes from, how can you be sure that
>> your program works at all? how can you be sure they're both strings?[/color]
>
> a and b are both string.[/color]

how do you know that?
[color=blue][color=green]
>> if you have unit tests, why don't they include Unicode tests?[/color]
>
> How do I structure the test cases to guarantee coverage? It is not practical to test every
> combinations of unicode/8-bit strings. Adding non-ascii characters to test data probably make
> problem pop up earlier. But it is arduous[/color]

sounds like you don't want to test for it. sorry, cannot help. I prefer
to design libraries so they can be tested, and design tests so they test all
important aspects of my libraries. if you prefer another approach, there's
not much I can do, other than repeating what I said at the start: if you do
things the right way (decode on the way in, encode on the way out), it
just works.

</F>

[Dieter Maurer]

Re: unicode encoding usablilty problem

"Fredrik Lundh" <fredrik@python ware.com> writes on Sat, 19 Feb 2005 18:44:27 +0100:[color=blue]
> "aurora" <aurora00@gmail .com> wrote:
>[color=green]
> > I don't want to mix them. But how could I find them? How do I know this statement can be
> > potential problem
> >
> > if a==b:
> >
> > where a and b can be instantiated individually far away from this line of code that put them
> > together?[/color][/color]

I do understand aurora's problems very well.

Me, too, I had suffered from this occasionally:

* some library decides to use unicode (without I had asked it to do so)

* Python decides then to convert other strings to unicode
and bum: "Unicode decode error".

I solve these issues with a "sys.setdefault encoding(ourDef aultEncoding)"
in "sitecustomize. py".

I know that almost all the characters I have to handle
are encoded in "ourDefaultEnco ding" and if something
converts to Unicode without being asked for, then this
is precisely the correct encoding.

I know that Unicode fanatists do not like "setdefaultenco ding"
but until we will have completely converted to Unicode (which we probably
will do in the farer future), this is essential to keep sane...


Dieter

[Martin v. Löwis]

Re: unicode encoding usablilty problem

aurora wrote:[color=blue]
> What is the processing of getting a PEP work out? Does the work and
> discussion carry out in the python-dev mailing list? I would be glad to
> help out especially on this particular issue.[/color]

See PEP 1 for the PEP process. The main point is that discussion is
*not* carried out on any specific forum. But instead, the PEP serves
as a container for all possible considerations people come up with,
formally by writing to the PEP author. Of course, they will use
comp.lang.pytho n and python-dev (and perhaps SIG mailing lists)
instead of writing to the PEP author, so the PEP author may need to
track these as well.

The process is triggered by the author posting revisions of the
PEP at a moderate rate, each time claiming "now I think it is
complete". Then, if nobody comes up with a reasoning that is
not yet covered in the PEP, it becomes ready for BDFL
pronouncement. It better also has an implementation at some point
in time.

For a dormant PEP, the prospective author should contact the
original author, and offer co-authoring. Perhaps the original
author even proposes that you can take over the entire thing
sometime.

Notice that, at some point, a patch implementing the PEP will
be needed. So you should indicate from the beginning whether you
are also willing to work on the implementation. If not, there is
a good chance that the PEP again goes dormant after the
specification is complete.

Regards,
Martin

[Vinay Sajip]

Re: unicode encoding usablilty problem

> This will help in your code, but there is big pile of modules in stdlib[color=blue]
> that are not unicode-friendly. From my daily practice come shlex
> (tokenizer works only with encoded strings) and logging (you cann't
> specify encoding for FileHandler).[/color]

You can, of course, pass in a stream opened using codecs.open to StreamHandler.
Not quite as friendly, I'll grant you.

Regards,


Vinay Sajip

[Pierre-Frédéric Caillaud]

HELP ! Anybody knows where the stackless python website is ?

Hello !

I've been trying desperately to access http://www.stackless.com but it's
been down, for about a week now !
I desperatly need to download stackless python...
Of course the stackless mailing list is on their server, so it's down,
too.

Does anybody has any info ?
Does anybody have a tarball of a recent version of stackless that I may
use (with the docs ?)

Thanks !

Regards,
P.F. Caillaud

[cfbolz]

Re: HELP ! Anybody knows where the stackless python website is ?

Hi!

Pierre-Frédéric Caillaud wrote:[color=blue]
> I've been trying desperately to access http://www.stackless.com but
> it's been down, for about a week now ![/color]

The stackless webpage is working again.

Regards,

Carl Friedrich Bolz