Bug or Feature?

[Stephan Diehl]

I was playing around with defining new types and have seen the
following behaviour:

Python 2.3.1
============
[color=blue][color=green][color=darkred]
>>> class Int(int):pass[/color][/color][/color]
....[color=blue][color=green][color=darkred]
>>> a = Int(7)
>>> b = Int(8)
>>> c = a + b
>>> type(c)[/color][/color][/color]
<type 'int'>

Basicly: Int is not closed under it's defined operations. :-(


by contrast:
[color=blue][color=green][color=darkred]
>>> class myset(Set):pass[/color][/color][/color]
....[color=blue][color=green][color=darkred]
>>> a = myset([1,2])
>>> b = myset([2,3])
>>> c = a & b
>>> type(c)[/color][/color][/color]
<class '__main__.myset '>

subclasses of Set are closed under the set operations. :-)

Should I consider this as a bug or something I have to live with?

Example, why this can be problematic:
Consider, I want to define an integer subtype that holds integers modulo N.

The straight forward way would be:

class mod7(int):
def __new__(cls,val ue):
return int.__new__(cls ,value%7)

Since addition, substraction, etc. doesn't work as expected, one must
redefine all relevant operators.
This is not what I'd consider object oriented behaviour.

Stephan

P.S.: for the mind police out there: Other than that, Python is great.

[Michele Simionato]

Re: Bug or Feature?

Stephan Diehl <stephan.diehl@ gmx.net> wrote in message news:<bpsik5$ni 7$07$1@news.t-online.com>...[color=blue]
> I was playing around with defining new types and have seen the
> following behaviour:
>
> Python 2.3.1
> ============
>[color=green][color=darkred]
> >>> class Int(int):pass[/color][/color]
> ...[color=green][color=darkred]
> >>> a = Int(7)
> >>> b = Int(8)
> >>> c = a + b
> >>> type(c)[/color][/color]
> <type 'int'>
>
> Basicly: Int is not closed under it's defined operations. :-(
>[/color]

You want my "innermetho ds" module. Originally I wrote it as a recipe
for the cookbook, but then I forgot to post it ;)

"""
A method of a class ``C`` is said to be an ``inner method`` of ``C``
if it returns an instance of ``C``. For instance, the ``.replace``
method of the string class is an inner method of ``str``, since
invoked by a string, it returns a string:
[color=blue][color=green][color=darkred]
>>> s='s'
>>> r=s.replace('s' ,'r')
>>> print r,type(r)[/color][/color][/color]
r <type 'str'>

In general, the property of being an inner method is not preserved
by inheritance. For instance, if I define a subclass of the string class,
[color=blue][color=green][color=darkred]
>>> class Str(str):[/color][/color][/color]
.... pass

then ``Str`` has still a ``.replace`` method (inherited from ``str``),
but it is not an inner method of ``Str``:
[color=blue][color=green][color=darkred]
>>> s=Str('s')
>>> r=s.replace('s' ,'r')
>>> print r,type(r) # the type of r is 'str', not 'Str'![/color][/color][/color]
r <type 'str'>

In many circumstances, this is not a problem. However, sometimes
it is important to keep the inner methods safe under inheritance.
If we are writing the methods from scratch, this can be done
by returning ``self.__class_ _(returnvalue)` ` instead of the
simple ``returnvalue`` . However, if the methods have been written
by others and we are simply inhering them, we need a more sophisticated
solution.

This module provides a ``protect_inner _methods`` metaclass-like function
which guarantees that the inner methods of the class invoking it are
safe under inheritance. As an exemplication and a commodoty, the
module also provides a ``Str`` class with protected versions of
all the inner methods of the builtin ``str`` class. The only
exception are ``__str__`` and ``__repr__``: it is better than
they stay unprotected, so the string representation of subclasses
of ``str`` is a simple plain string, not an enhanced one. In
this way ``str(x)`` continues to convert ``x`` instances in ``str``
instances. If ``Str`` instances are wanted, use ``Str()``!

Here is an example, showing that ``Str.replace`` is an inner method
of ``Str``:
[color=blue][color=green][color=darkred]
>>> from innermethods import Str
>>> s=Str('s')
>>> r=s.replace('s' ,'r') # .replace invoked from an instance of 'Str'
>>> print r,type(r) # returns an instance of 'Str', i.e. it is inner method[/color][/color][/color]
r <class 'innermethods.S tr'>

Let me show that the inner methods of ``Str`` are protected under
inheritance, i.e. they continues to be inner methods of the
subclasses:
[color=blue][color=green][color=darkred]
>>> class MyStr(Str):[/color][/color][/color]
.... pass
[color=blue][color=green][color=darkred]
>>> s=MyStr('s')
>>> r=s.replace('s' ,'r') # .replace invoked from an instance of 'Str'
>>> print r,type(r) # returns an instance of 'Str', i.e. it is inner method[/color][/color][/color]
r <class 'MyStr'>

"""

def protect_inner__ method(cls,meth name):
"""Given a class and an inner method name, returns a wrapped
version of the inner method. Raise an error if the method is not
defined in any ancestors of the class. In the case of duplicated
names, the first method in the MRO is taken."""
def _(self,*args,** kw):
supermethod=get attr(super(cls, self),methname)
return self.__class__( supermethod(*ar gs,**kw))
return _

def protect_inner_m ethods(name,bas es,dic):
"""Metaclas s-like function. Returns a new class with inner methods
protected under inheritance. The calling class must provide a
``__innermethod s__`` attribute containing the list of the
method names to be protected, otherwise nothing will be done."""
cls=type(name,b ases,dic) # generate the new class
innermethods=di c.get('__innerm ethods__',[])
# look for the list of methods to wrap and wrap them all
for methname in innermethods:
setattr(cls,met hname,protect_i nner__method(cl s,methname))
return cls

class Str(str):
"""A string class with wrapped inner methods."""
__metaclass__=p rotect_inner_me thods
__innermethods_ _="""__add__ __mod__ __mul__ __rmod__ __rmul__ capitalize
center expandtabs join ljust lower lstrip replace rjust rstrip strip
swapcase title translate upper zfill""".split( )
def __radd__(self,o ther): # inner method
"""Guarante es ``'a'+Str('b')` ` returns a ``Str`` instance.
Also ``s='s'; s+=Str('b')`` makes ``s`` a ``Str`` instance."""
return self.__class__( other.__add__(s elf))

[Stephan Diehl]

Re: Bug or Feature?

Michele Simionato wrote:
[color=blue]
> Stephan Diehl <stephan.diehl@ gmx.net> wrote in message
> news:<bpsik5$ni 7$07$1@news.t-online.com>...[color=green]
>> I was playing around with defining new types and have seen the
>> following behaviour:
>>
>> Python 2.3.1
>> ============
>>[color=darkred]
>> >>> class Int(int):pass[/color]
>> ...[color=darkred]
>> >>> a = Int(7)
>> >>> b = Int(8)
>> >>> c = a + b
>> >>> type(c)[/color]
>> <type 'int'>
>>
>> Basicly: Int is not closed under it's defined operations. :-(
>>[/color]
>
> You want my "innermetho ds" module. Originally I wrote it as a recipe
> for the cookbook, but then I forgot to post it ;)
>[/color]

Yes, thanks, this is most helpfull.

Although, the question was more along the line, if (in the light of the
principle of least surprise) this behaviour makes sense.
My private opinion would be, that the usefullness of builtin types as real
types would be much better, if they were safe under inheritance.

[Michael Hudson]

Re: Bug or Feature?

Stephan Diehl <stephan.diehl@ gmx.net> writes:
[color=blue]
> Although, the question was more along the line, if (in the light of the
> principle of least surprise) this behaviour makes sense.
> My private opinion would be, that the usefullness of builtin types as real
> types would be much better, if they were safe under inheritance.[/color]

The problem is that it's impossible for the int type to know about the
requirements of your subclass' constructor. Explicit is better than
implicit, and all that.

For the example you posted, I don't see much of an advantage to
inheriting from int.

Cheers,
mwh

--
If a train station is a place where a train stops, what's a
workstation? -- unknown (to me, at least)

[Duncan Booth]

Re: Bug or Feature?

Stephan Diehl <stephan.diehl@ gmx.net> wrote in
news:bpt5do$nls $04$1@news.t-online.com:
[color=blue]
> Although, the question was more along the line, if (in the light of
> the principle of least surprise) this behaviour makes sense.
> My private opinion would be, that the usefullness of builtin types as
> real types would be much better, if they were safe under inheritance.
>[/color]
There are some tradeoffs here.

I think that the majority of users would say that the speed of operations
using 'int' is more important than the ease of inheriting from them. If you
can make the 'int' operations work this way, but keep them at least as fast
as they are today, then there may be a case for changing the behaviour.

The other problem though is that this would break backwards compatibility.
For example if any code were to currently add booleans, it would suddenly
get a boolean result. The (untested) code below, currently counts how many
items in a list are greater than a specified value, but under your scheme
it would return a boolean.

def HowManyGreaterT han(aList, aValue):
return reduce(operator .add, [ x > aValue for x in aList ])

I think what would be most useful would be new subclasses of int etc. which
have the properties you desire, but you can write them yourself easily
enough.

What rules, BTW, are you proposing for operations between two separate
subclasses of int? e.g. given your mod7 class, and a corresponding mod9
class should this print 3, 6, 24 or throw an exception?

x = mod7(3)
y = mod9(8)
print x*y

and does this do the same?

print y*x

--
Duncan Booth duncan@rcp.co.u k
int month(char *p){return(1248 64/((p[0]+p[1]-p[2]&0x1f)+1)%12 )["\5\x8\3"
"\6\7\xb\1\x9\x a\2\0\4"];} // Who said my code was obscure?

[Stephan Diehl]

Re: Bug or Feature?

Duncan Booth wrote:
[color=blue]
> Stephan Diehl <stephan.diehl@ gmx.net> wrote in
> news:bpt5do$nls $04$1@news.t-online.com:
>[color=green]
>> Although, the question was more along the line, if (in the light of
>> the principle of least surprise) this behaviour makes sense.
>> My private opinion would be, that the usefullness of builtin types as
>> real types would be much better, if they were safe under inheritance.
>>[/color]
> There are some tradeoffs here.
>
> I think that the majority of users would say that the speed of operations
> using 'int' is more important than the ease of inheriting from them. If
> you can make the 'int' operations work this way, but keep them at least as
> fast as they are today, then there may be a case for changing the
> behaviour.[/color]

O.K., since the stuff I'm concerned about here is of no relevance for
99.9 % of all python programmers, execution speed is a very valid reason for
this (and I'll consider this behaviour as a feature :-)
[color=blue]
>
> The other problem though is that this would break backwards compatibility.
> For example if any code were to currently add booleans, it would suddenly
> get a boolean result. The (untested) code below, currently counts how many
> items in a list are greater than a specified value, but under your scheme
> it would return a boolean.
>
> def HowManyGreaterT han(aList, aValue):
> return reduce(operator .add, [ x > aValue for x in aList ])[/color]

Which, of course, would bring us to the discussion if a bool should be a
subtype of int (and since this was decided long ago, it should not be
discussed, of course)
[color=blue]
>
> I think what would be most useful would be new subclasses of int etc.
> which have the properties you desire, but you can write them yourself
> easily enough.
>
> What rules, BTW, are you proposing for operations between two separate
> subclasses of int? e.g. given your mod7 class, and a corresponding mod9
> class should this print 3, 6, 24 or throw an exception?
>
> x = mod7(3)
> y = mod9(8)
> print x*y
>
> and does this do the same?
>
> print y*x
>[/color]

tough question. I'd probably throw an exception.

thanks for your input

Stephan

[Stephan Diehl]

Re: Bug or Feature?

Michael Hudson wrote:
[color=blue]
> Stephan Diehl <stephan.diehl@ gmx.net> writes:
>[color=green]
>> Although, the question was more along the line, if (in the light of the
>> principle of least surprise) this behaviour makes sense.
>> My private opinion would be, that the usefullness of builtin types as
>> real types would be much better, if they were safe under inheritance.[/color]
>
> The problem is that it's impossible for the int type to know about the
> requirements of your subclass' constructor. Explicit is better than
> implicit, and all that.
>
> For the example you posted, I don't see much of an advantage to
> inheriting from int.[/color]

This is probably more a theoretical question, but if I decide to have an
operation within one specific domain, I want the result in that domain too.
Otherwise, there would be no point at all to define such a numeric class at
all.

Stephan[color=blue]
>
> Cheers,
> mwh
>[/color]

[Mel Wilson]

Re: Bug or Feature?

In article <m3llq5zufq.fsf @pc150.maths.br is.ac.uk>,
Michael Hudson <mwh@python.net > wrote:[color=blue]
>Stephan Diehl <stephan.diehl@ gmx.net> writes:
>[color=green]
>> Although, the question was more along the line, if (in the light of the
>> principle of least surprise) this behaviour makes sense.
>> My private opinion would be, that the usefullness of builtin types as real
>> types would be much better, if they were safe under inheritance.[/color]
>
>The problem is that it's impossible for the int type to know about the
>requirements of your subclass' constructor. Explicit is better than
>implicit, and all that.[/color]

Not strictly true. A numeric operation can access the
objects class, as:

def __mul__ (self, other):
if not isinstance (other, Decimal):
other = Decimal (other)
return self.__class__ (self._v * other._v, self._e + other._e)

so that operations on a descendant of the Decimal class will
return objects of the descendants class. (Strange that I
didn't code `isinstance (other, self.__class__) `, etc. Must
think out the consequences of that.)

What this would mean in runtime for the bazillions of int
operations that run every day, I don't know.
[color=blue]
>For the example you posted, I don't see much of an advantage to
>inheriting from int.[/color]

True. With int working as it does, it might be better to
pull out all the stops and write an Int class that supports
full inheritance, and work from there.

Regards. Mel.

[Dang Griffith]

Re: Bug or Feature?

On Mon, 24 Nov 2003 18:13:34 +0100, Stephan Diehl
<stephan.diehlN OSPAM@gmx.net> wrote:

....[color=blue]
>This is probably more a theoretical question, but if I decide to have an
>operation within one specific domain, I want the result in that domain too.
>Otherwise, there would be no point at all to define such a numeric class at
>all.
>
>Stephan[/color]

I understand what you're saying, but there's no universal rule that
says that the result of operations between members of a set are also
members of the set. Obvious examples include division on the set of
integers, and division over the set of real numbers, i.e.division by
zero is not a real number.
--dang

[Michael Hudson]

Re: Bug or Feature?

mwilson@the-wire.com (Mel Wilson) writes:
[color=blue]
> In article <m3llq5zufq.fsf @pc150.maths.br is.ac.uk>,
> Michael Hudson <mwh@python.net > wrote:[color=green]
> >Stephan Diehl <stephan.diehl@ gmx.net> writes:
> >[color=darkred]
> >> Although, the question was more along the line, if (in the light of the
> >> principle of least surprise) this behaviour makes sense.
> >> My private opinion would be, that the usefullness of builtin types as real
> >> types would be much better, if they were safe under inheritance.[/color]
> >
> >The problem is that it's impossible for the int type to know about the
> >requirements of your subclass' constructor. Explicit is better than
> >implicit, and all that.[/color]
>
> Not strictly true.[/color]

Um, did you actually read what I said? Again, with emphasis:
[color=blue][color=green]
> >The problem is that it's impossible for the int type to know about the
> >*REQUIREMENT S OF YOUR SUBCLASS' CONSTRUCTOR*.[/color][/color]
[color=blue]
> A numeric operation can access the objects class, as:
>
> def __mul__ (self, other):
> if not isinstance (other, Decimal):
> other = Decimal (other)
> return self.__class__ (self._v * other._v, self._e + other._e)
>
> so that operations on a descendant of the Decimal class will
> return objects of the descendants class.[/color]

Yes, but now you're constrainting future subclasses' constructors.
[color=blue]
> What this would mean in runtime for the bazillions of int
> operations that run every day, I don't know.[/color]

There would be ways to avoid that, I think.
[color=blue][color=green]
> >For the example you posted, I don't see much of an advantage to
> >inheriting from int.[/color]
>
> True. With int working as it does, it might be better to
> pull out all the stops and write an Int class that supports
> full inheritance, and work from there.[/color]

I think someone has written such a UserInt somewhere...

It seems to be generally good advice to only subclass builtin types in
Python if you want to ADD behaviour, not change it. Not 100% true,
but close.

Cheers,
mwh

--
LINTILLA: You could take some evening classes.
ARTHUR: What, here?
LINTILLA: Yes, I've got a bottle of them. Little pink ones.
-- The Hitch-Hikers Guide to the Galaxy, Episode 12

[Stephan Diehl]

Re: Bug or Feature?

Dang Griffith wrote:
[color=blue]
> On Mon, 24 Nov 2003 18:13:34 +0100, Stephan Diehl
> <stephan.diehlN OSPAM@gmx.net> wrote:
>
> ...[color=green]
>>This is probably more a theoretical question, but if I decide to have an
>>operation within one specific domain, I want the result in that domain
>>too. Otherwise, there would be no point at all to define such a numeric
>>class at all.
>>
>>Stephan[/color]
>
> I understand what you're saying, but there's no universal rule that
> says that the result of operations between members of a set are also
> members of the set. Obvious examples include division on the set of
> integers, and division over the set of real numbers, i.e.division by
> zero is not a real number.
> --dang[/color]

If we have a look at the mathematical definition of numbers, the interesting
thing is not the set of these numbers, but the set combined with some
usefull operators (like addition and multiplication) .
It is (mathematicaly speaking) not possible to have a result under these
operations that are outside the definition.

In that sense, there IS this universal rule, you were talking about. (and
division by zero is just not allowed by definition).

Stephan

[Dang Griffith]

Re: Bug or Feature?

On Mon, 24 Nov 2003 19:35:51 +0100, Stephan Diehl
<stephan.diehlN OSPAM@gmx.net> wrote:
[color=blue]
>Dang Griffith wrote:
>[color=green]
>> On Mon, 24 Nov 2003 18:13:34 +0100, Stephan Diehl
>> <stephan.diehlN OSPAM@gmx.net> wrote:
>>
>> ...[color=darkred]
>>>This is probably more a theoretical question, but if I decide to have an
>>>operation within one specific domain, I want the result in that domain
>>>too. Otherwise, there would be no point at all to define such a numeric
>>>class at all.
>>>
>>>Stephan[/color]
>>
>> I understand what you're saying, but there's no universal rule that
>> says that the result of operations between members of a set are also
>> members of the set. Obvious examples include division on the set of
>> integers, and division over the set of real numbers, i.e.division by
>> zero is not a real number.
>> --dang[/color]
>
>If we have a look at the mathematical definition of numbers, the interesting
>thing is not the set of these numbers, but the set combined with some
>usefull operators (like addition and multiplication) .
>It is (mathematicaly speaking) not possible to have a result under these
>operations that are outside the definition.
>
>In that sense, there IS this universal rule, you were talking about. (and
>division by zero is just not allowed by definition).
>
>Stephan[/color]
And for the set of whole numbers, some divisions are allowed, but any
division that results in a fraction would not be allowed. By
definition, i.e. a combination of operators and operands that result
in a value not in the same domain is not a valid combination. It
works, but seems circular. :-)
--dang

[Stephan Diehl]

Re: Bug or Feature?

Dang Griffith wrote:
[color=blue]
> On Mon, 24 Nov 2003 19:35:51 +0100, Stephan Diehl
> <stephan.diehlN OSPAM@gmx.net> wrote:
>[color=green]
>>Dang Griffith wrote:
>>[color=darkred]
>>> On Mon, 24 Nov 2003 18:13:34 +0100, Stephan Diehl
>>> <stephan.diehlN OSPAM@gmx.net> wrote:
>>>
>>> ...
>>>>This is probably more a theoretical question, but if I decide to have an
>>>>operation within one specific domain, I want the result in that domain
>>>>too. Otherwise, there would be no point at all to define such a numeric
>>>>class at all.
>>>>
>>>>Stephan
>>>
>>> I understand what you're saying, but there's no universal rule that
>>> says that the result of operations between members of a set are also
>>> members of the set. Obvious examples include division on the set of
>>> integers, and division over the set of real numbers, i.e.division by
>>> zero is not a real number.
>>> --dang[/color]
>>
>>If we have a look at the mathematical definition of numbers, the
>>interesting thing is not the set of these numbers, but the set combined
>>with some usefull operators (like addition and multiplication) .
>>It is (mathematicaly speaking) not possible to have a result under these
>>operations that are outside the definition.
>>
>>In that sense, there IS this universal rule, you were talking about. (and
>>division by zero is just not allowed by definition).
>>
>>Stephan[/color]
> And for the set of whole numbers, some divisions are allowed, but any
> division that results in a fraction would not be allowed. By
> definition, i.e. a combination of operators and operands that result
> in a value not in the same domain is not a valid combination. It
> works, but seems circular. :-)
> --dang[/color]

You might have noticed, I was talking about Mathematics, not the imperfect
implementation of it in computers :-)
To have division as a meaningfull operation, the set in question must be a
group under the '*' operation (See for example
"http://en2.wikipedia.o rg/wiki/Group_(mathemat ics)" )
The point is, that division is the reverse operation to multiplication.

Cheers, Stephan

[Gonçalo Rodrigues]

Re: Bug or Feature?

On Tue, 25 Nov 2003 16:04:57 +0100, Stephan Diehl
<stephan.diehlN OSPAM@gmx.net> wrote:

[snip]
[color=blue]
>
>You might have noticed, I was talking about Mathematics, not the imperfect
>implementati on of it in computers :-)
>To have division as a meaningfull operation, the set in question must be a
>group under the '*' operation (See for example
>"http://en2.wikipedia.o rg/wiki/Group_(mathemat ics)" )
>The point is, that division is the reverse operation to multiplication.[/color]

But when talking of "numbers" a group is not enough, you want a field
- a set with two operations (in some situations something weaker, like
integral domains, is enough). In fields, such as real numbers, complex
numbers, p mod integers, p-adic fields, etc. every number has a
multiplicative inverse *except* the zero.

Pedantically yours,
G. Rodrigues

[Patrick Maupin]

Re: Bug or Feature?

Michael Hudson wrote:
[color=blue]
> I think someone has written such a UserInt somewhere...[/color]

There have been a couple of UserInts floating around.
I think the most recent was mine.

The bad news is that my UserInt module actually returns
ints for all operations.

The good news is that, being a lazy programmer, I didn't
actually write my "UserInt" module -- I let my computer
do it for me, by writing a script which would generate
the module.

The main reason I posted the script is because it can
easily be adapted to create any sort of UserXXX classes,
as well as to add any sort of behavior mods to those
classes. I think it would be trivial to create a UserInt
class which returns other instances of itself for arithmetic
operations by using the script:

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=mailman.85.1066086131.2192.python-list%40python.org

Regards,
Pat