unicode mystery

[Sean McIlroy]

I recently found out that unicode("\347", "iso-8859-1") is the
lowercase c-with-cedilla, so I set out to round up the unicode numbers
of the extra characters you need for French, and I found them all just
fine EXCEPT for the o-e ligature (oeuvre, etc). I examined the unicode
characters from 0 to 900 without finding it; then I looked at
www.unicode.org but the numbers I got there (0152 and 0153) didn't
work. Can anybody put a help on me wrt this? (Do I need to give a
different value for the second parameter, maybe?)

Peace,
STM

PS: I'm considering looking into pyscript as a means of making
diagrams for inclusion in LaTeX documents. If anyone can share an
opinion about pyscript, I'm interested to hear it.

Peace

[John Lenton]

Re: unicode mystery

On Mon, Jan 10, 2005 at 07:48:44PM -0800, Sean McIlroy wrote:[color=blue]
> I recently found out that unicode("\347", "iso-8859-1") is the
> lowercase c-with-cedilla, so I set out to round up the unicode numbers
> of the extra characters you need for French, and I found them all just
> fine EXCEPT for the o-e ligature (oeuvre, etc). I examined the unicode
> characters from 0 to 900 without finding it; then I looked at
> www.unicode.org but the numbers I got there (0152 and 0153) didn't
> work. Can anybody put a help on me wrt this? (Do I need to give a
> different value for the second parameter, maybe?)[/color]

Å“ isn't part of ISO 8859-1, so you can't get it that way. You can do
one of

u'\u0153'

or, if you must,

unicode("\305\2 23", "utf-8")

--
John Lenton (john@grulic.or g.ar) -- Random fortune:
Lisp, Lisp, Lisp Machine,
Lisp Machine is Fun.
Lisp, Lisp, Lisp Machine,
Fun for everyone.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFB42K4gPq u395ykGsRAuYHAK CWQPoNdtAaBm6Xe KqN4/cdsVIhJgCggMRq
NlFH8U/HGRTNkYrZsFCulV g=
=47J7
-----END PGP SIGNATURE-----

[John Machin]

Re: unicode mystery


Sean McIlroy wrote:[color=blue]
> I recently found out that unicode("\347", "iso-8859-1") is the
> lowercase c-with-cedilla, so I set out to round up the unicode[/color]
numbers[color=blue]
> of the extra characters you need for French, and I found them all[/color]
just[color=blue]
> fine EXCEPT for the o-e ligature (oeuvre, etc). I examined the[/color]
unicode[color=blue]
> characters from 0 to 900 without finding it; then I looked at
> www.unicode.org but the numbers I got there (0152 and 0153) didn't
> work. Can anybody put a help on me wrt this? (Do I need to give a
> different value for the second parameter, maybe?)[/color]

Characters that are in iso-8859-1 are mapped directly into Unicode.
That is, the first 256 characters of Unicode are identical to
iso-8859-1.

Consider this:
[color=blue][color=green][color=darkred]
>>> c_cedilla = unicode("\347", "iso-8859-1")
>>> c_cedilla[/color][/color][/color]
u'\xe7'[color=blue][color=green][color=darkred]
>>> ord(c_cedilla)[/color][/color][/color]
231[color=blue][color=green][color=darkred]
>>> ord("\347")[/color][/color][/color]
231

What you did with c_cedilla "worked" because it was effectively doing
nothing. However if you do unicode(char, encoding) where char is not in
encoding, it won't "work".

As John Lenton has pointed out, if you find a character in the Unicode
tables, you can just use it directly. There is no need in this
circumstance to use unicode().

HTH,
John

[John Machin]

Octal notation: severe deprecation


Some poster wrote (in connexion with another topic):[color=blue]
> ... unicode("\347", "iso-8859-1") ...[/color]

Well, I haven't had a good rant for quite a while, so here goes:

I'm a bit of a retro specimen, being able (inter alia) to recall octal
opcodes from the ICT 1900 series (070=call, 072=exit, 074=branch, ...)
but nowadays I regard continued usage of octal as a pox and a
pestilence.

1. Octal notation is of use to systems programmers on computers where
the number of bits in a word is a multiple of 3. Are there any still in
production use? AFAIK word sizes were 12, 24, 36, 48, and 60 bits --
all multiples of 4, so hexadecimal could be used.

2. Consider the effect on the newbie who's never even heard of "octal":
[color=blue][color=green][color=darkred]
>>> import datetime
>>> datetime.date(2 005,01,01)[/color][/color][/color]
datetime.date(2 005, 1, 1)[color=blue][color=green][color=darkred]
>>> datetime.date(2 005,09,09)[/color][/color][/color]
File "<stdin>", line 1
datetime.date(2 005,09,09)
^
SyntaxError: invalid token

[straight out of the "BOFH Manual of Po-faced Error Messages"]

3. Consider this extract from the docs for the re module:
"""
\number
Matches the contents of the group of the same number. Groups are
numbered starting from 1. For example, (.+) \1 matches 'the the' or '55
55', but not 'the end' (note the space after the group). This special
sequence can only be used to match one of the first 99 groups. If the
first digit of number is 0, or number is 3 octal digits long, it will
not be interpreted as a group match, but as the character with octal
value number. Inside the "[" and "]" of a character class, all numeric
escapes are treated as characters.
"""

I helped to straighten out this description a few years ago, but I fear
it's still not 100% accurate. Worse, take a peek at the code necessary
to implement this.

===

We (un-Pythonically) implicitly take a leading zero (or even just
\[0-7]) as meaning octal, instead of requiring something explicit as
with hexadecimal. The variable length idea in strings doesn't help:
"\18", "\128" and "\1238" are all strings of length 2.

I don't see any mention of octal in GvR's "Python Regrets" or AMK's
"PEP 3000". Why not? Is it not regretted?

[PJDM]

Re: Octal notation: severe deprecation

John Machin wrote:[color=blue]
>
> 1. Octal notation is of use to systems programmers on computers where
> the number of bits in a word is a multiple of 3. Are there any still[/color]
in[color=blue]
> production use? AFAIK word sizes were 12, 24, 36, 48, and 60 bits --
> all multiples of 4, so hexadecimal could be used.[/color]

The PDP-11 was 16 bit, but used octal. With eight registers and eight
addressing modes in instructions, octal was a convenient base. (On the
11/70, the front switches were marked in alternate pink and purple
groups of three. <http://www.psych.usyd. edu.au/pdp-11/11_70.html>)
[color=blue]
> I don't see any mention of octal in GvR's "Python Regrets" or AMK's
> "PEP 3000". Why not? Is it not regretted?[/color]

I suspect that, as in other places, Python just defers to the
underlying implementation. Since the C libraries treat "0n" as octal,
so does Python. (So does JavaScript.) More "it sucks, but it's too late
to change" than regrets.

Maybe P3K will have an integer literal like "n_b" for "the integer n in
base b".

PJDM

[Stephen Thorne]

Re: Octal notation: severe deprecation

On 12 Jan 2005 16:21:29 -0800, PJDM <PeterMayne@ap. spherion.com> wrote:[color=blue]
> Maybe P3K will have an integer literal like "n_b" for "the integer n in
> base b".[/color]

I would actually like to see pychecker pick up conceptual errors like this:

import datetime
datetime.dateti me(2005, 04,04)

Regards,
Stephen Thorne

[and-google@doxdesk.com]

Re: Octal notation: severe deprecation

John Machin wrote:
[color=blue]
> I regard continued usage of octal as a pox and a pestilence.[/color]

Quite agree. I was disappointed that it ever made it into Python.

Octal's only use is:

a) umasks
b) confusing the hell out of normal non-programmers for whom a
leading zero is in no way magic

(a) does not outweigh (b).

In Mythical Future Python I would like to be able to use any base in
integer literals, which would be better. Example random syntax:

flags= 2x0001101010100 1
umask= 8x664
answer= 10x42
addr= 16x0E800004 # 16x == 0x
gunk= 36x8H6Z9A0X

But either way, I want rid of 0->octal.
[color=blue]
> Is it not regretted?[/color]

Maybe the problem just doesn't occur to people who have used C too
long.

OT: Also, if Google doesn't stop lstrip()ing my posts I may have to get
a proper news feed. What use is that on a Python newsgroup? Grr.
--
Andrew Clover
mailto:and@doxd esk.com

http://www.doxdesk.com/

[Tim Roberts]

Re: Octal notation: severe deprecation

Stephen Thorne <stephen.thorne @gmail.com> wrote:
[color=blue]
>On 12 Jan 2005 16:21:29 -0800, PJDM <PeterMayne@ap. spherion.com> wrote:[color=green]
>> Maybe P3K will have an integer literal like "n_b" for "the integer n in
>> base b".[/color]
>
>I would actually like to see pychecker pick up conceptual errors like this:
>
>import datetime
>datetime.datet ime(2005, 04,04)[/color]

Why is that a conceptual error? Syntactically, this could be a valid call
to a function. Even if you have parsed and executed datetime, so that you
know datetime.dateti me is a class, it's quite possible that the creation
and destruction of an object might have useful side effects.
--
- Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

[Peter Hansen]

Re: Octal notation: severe deprecation

and-google@doxdesk. com wrote:[color=blue]
> In Mythical Future Python I would like to be able to use any base in
> integer literals, which would be better. Example random syntax:
>
> flags= 2x0001101010100 1
> umask= 8x664
> answer= 10x42
> addr= 16x0E800004 # 16x == 0x
> gunk= 36x8H6Z9A0X[/color]

I think I kinda like this idea. Allowing arbitrary values,
however, would probably be pointless, as there are very
few bases in common enough use that a language should make
it easy to write literals in any of them. So I think "36x"
is silly, and would suggest limiting this to 2, 8, 10, and
16. At the very least, a range of 2-16 should be used.
(It would be cute but pointless to allow 1x000000000. :-)

-Peter

[phil_nospam_schmidt@yahoo.com]

Re: Octal notation: severe deprecation

Peter Hansen wrote:[color=blue]
> and-google@doxdesk. com wrote:[color=green]
> > In Mythical Future Python I would like to be able to use any base[/color][/color]
in[color=blue][color=green]
> > integer literals, which would be better. Example random syntax:
> >
> > flags= 2x0001101010100 1
> > umask= 8x664
> > answer= 10x42
> > addr= 16x0E800004 # 16x == 0x
> > gunk= 36x8H6Z9A0X[/color]
>
> I think I kinda like this idea. Allowing arbitrary values,
> however, would probably be pointless, as there are very
> few bases in common enough use that a language should make
> it easy to write literals in any of them. So I think "36x"
> is silly, and would suggest limiting this to 2, 8, 10, and
> 16. At the very least, a range of 2-16 should be used.
> (It would be cute but pointless to allow 1x000000000. :-)
>
> -Peter
>
>
>[/color]
How about base 24 and 60, for hours and minutes/seconds?

[Steve Holden]

Re: Octal notation: severe deprecation

and-google@doxdesk. com wrote:
[color=blue]
> John Machin wrote:
>
>[color=green]
>>I regard continued usage of octal as a pox and a pestilence.[/color]
>
>
> Quite agree. I was disappointed that it ever made it into Python.
>
> Octal's only use is:
>
> a) umasks
> b) confusing the hell out of normal non-programmers for whom a
> leading zero is in no way magic
>
> (a) does not outweigh (b).
>
> In Mythical Future Python I would like to be able to use any base in
> integer literals, which would be better. Example random syntax:
>
> flags= 2x0001101010100 1
> umask= 8x664
> answer= 10x42
> addr= 16x0E800004 # 16x == 0x
> gunk= 36x8H6Z9A0X
>
> But either way, I want rid of 0->octal.
>
>[color=green]
>>Is it not regretted?[/color]
>
>
> Maybe the problem just doesn't occur to people who have used C too
> long.
>[/color]
:-)
[color=blue]
> OT: Also, if Google doesn't stop lstrip()ing my posts I may have to get
> a proper news feed. What use is that on a Python newsgroup? Grr.[/color]

I remember using a langauge (Icon?) in which arbitrary bases up to 36
could be used with numeric literals. IIRC, the literals had to begin
with the base in decimnal, folowed by a "b" followed by the digits of
the value using a through z for digits from ten to thirty-five. So

gunk = 36b8H6Z9A0X

would have been valid.

nothing-new-under-the-sun-ly y'rs - steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119

[Dan Sommers]

Re: Octal notation: severe deprecation

On Thu, 13 Jan 2005 09:56:15 -0500,
Steve Holden <steve@holdenwe b.com> wrote:
[color=blue]
> I remember using a langauge (Icon?) in which arbitrary bases up to 36
> could be used with numeric literals. IIRC, the literals had to begin
> with the base in decimnal, folowed by a "b" followed by the digits of
> the value using a through z for digits from ten to thirty-five. So[/color]
[color=blue]
> gunk = 36b8H6Z9A0X[/color]
[color=blue]
> would have been valid.[/color]

Lisp also allows for literals in bases from 2 through 36.

Lisp also allows programs to change the default (away from decimal), so
that an "identifier " like aa is read by the parser as a numeric constant
with the decimal value of 170. Obviously, this has to be used with
care, but makes reading external data files written in strange bases
very easy.
[color=blue]
> nothing-new-under-the-sun-ly y'rs - steve[/color]

every-language-wants-to-be-lisp-ly y'rs,
Dan

--
Dan Sommers
<http://www.tombstoneze ro.net/dan/>
Never play leapfrog with a unicorn.

[Leif K-Brooks]

Re: Octal notation: severe deprecation

Tim Roberts wrote:[color=blue]
> Stephen Thorne <stephen.thorne @gmail.com> wrote:
>[color=green]
>>I would actually like to see pychecker pick up conceptual errors like this:
>>
>>import datetime
>>datetime.date time(2005, 04,04)[/color]
>
>
> Why is that a conceptual error? Syntactically, this could be a valid call
> to a function. Even if you have parsed and executed datetime, so that you
> know datetime.dateti me is a class, it's quite possible that the creation
> and destruction of an object might have useful side effects.[/color]

I'm guessing that Stephen is saying that PyChecker should have special
knowledge of the datetime module and of the fact that dates are often
specified with a leading zero, and therefor complain that they shouldn't
be used that way in Python source code.

[Bengt Richter]

Re: Octal notation: severe deprecation

On Thu, 13 Jan 2005 08:18:25 -0500, Peter Hansen <peter@engcorp. com> wrote:
[color=blue]
>and-google@doxdesk. com wrote:[color=green]
>> In Mythical Future Python I would like to be able to use any base in
>> integer literals, which would be better. Example random syntax:
>>
>> flags= 2x0001101010100 1
>> umask= 8x664
>> answer= 10x42
>> addr= 16x0E800004 # 16x == 0x
>> gunk= 36x8H6Z9A0X[/color]
>
>I think I kinda like this idea. Allowing arbitrary values,
>however, would probably be pointless, as there are very
>few bases in common enough use that a language should make
>it easy to write literals in any of them. So I think "36x"
>is silly, and would suggest limiting this to 2, 8, 10, and
>16. At the very least, a range of 2-16 should be used.
>(It would be cute but pointless to allow 1x000000000. :-)
>[/color]
My concern is negative numbers when you are interested in the
bits of a typical twos-complement number. (BTW, please don't tell me
that's platform-specific hardware oriented stuff: Two's complement is
a fine abstraction for interpreting a bit vector, which is another
fine abstraction ;-)

One way to do it consistently is to have a sign digit as the first
digit after the x, which is either 0 or base-1 -- e.g., +3 and -3 would be

2x011 2x101
8x03 8x75
16x03 16xfd
10x03 10x97

Then the "sign digit" can be extended indefinitely to the left without
changing the value, noting that -3 == 97-100 == 997-1000) and similarly
for other bases: -3 == binary 101-1000 == 111101-1000000 etc. IOW, you just
subtract base**<number of digits in representation> if the first digit
is base-1, to get the negative value.

This would let us have a %<width>.<base> b format to generate the digits part
and would get around the ugly hack for writing hex literals of negative numbers.

def __repr__(self): return '<%s object at %08.16b>' %(type(self).__ name__, id(self))

and you could write based literals in the above formats with e.g., with
'16x%.16b' % number
or
'2x%.2b' % number
etc.

Regards,
Bengt Richter

[Jeff Epler]

Re: Octal notation: severe deprecation

On Thu, Jan 13, 2005 at 11:04:21PM +0000, Bengt Richter wrote:[color=blue]
> One way to do it consistently is to have a sign digit as the first
> digit after the x, which is either 0 or base-1 -- e.g., +3 and -3 would be
>
> 2x011 2x101
> 8x03 8x75
> 16x03 16xfd
> 10x03 10x97[/color]

... so that 0x8 and 16x8 would be different? So that 2x1 and 2x01 would
be different?

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFB5weEJd0 1MZaTXX0RAvDCAJ 46GzXpJobv6dSiI KNGmyelRerFPwCf XmRs
9i51i1LelXbZO26 izFwYv58=
=vjU0
-----END PGP SIGNATURE-----