Class Variable Inheritance

[Brian Jones]

I'm sure the solution may be obvious, but this problem is driving me
mad. The following is my code:

class a(object):

mastervar = []

def __init__(self):
print 'called a'

class b(a):

def __init__(self):
print 'called b'
self.mapvar()

def mapvar(self):
self.mastervar. append(['b'])

class c(b):

def __init__(self):
print 'called c'
self.mapvar()

def mapvar(self):
super(c, self).mapvar()
self.mastervar. append(['c'])

if __name__ == '__main__':

a1 = a()
b1 = b()
c1 = c()
d1 = c() # Call C again

print a1.mastervar
print b1.mastervar
print c1.mastervar
print d1.mastervar

What I don't understand is why mastervar gets modified by each _seperate
instance_ of classes that happen to extend the base class 'a'.
Shouldn't mastervar be contained within the scope of the inheriting
classes? Why is it being treated like a global variable and being
modified by the other instances?


Thanks,

Brian "bojo" Jones

[Brian \bojo\ Jones]

Re: Class Variable Inheritance

It became clear to me that mastervar inside of class a is a static
variable, and is associated with all instances of classes that extend
class a. To get around this, I implemented a seperate mapping class:

class mapper:

def __init__(self):
self.mastermap = []

def add(self, map):
self.mastermap. append(map)

def get(self):
return self.mastermap

class a(object):

def __init__(self):
self.map = mapper()
print 'called a'

class b(a):

def __init__(self):
self.map = mapper()
print 'called b'
self.mapvar()

def mapvar(self):
self.map.add('b ')

class c(b):

def __init__(self):
self.map = mapper()
print 'called c'
self.mapvar()

def mapvar(self):
super(c, self).mapvar()
self.map.add('c ')

if __name__ == '__main__':

a1 = a()
a2 = a()
b1 = b()
c1 = c()
d1 = c() # Call C again

print a1.map.get()
print a1.map.get()
print b1.map.get()
print c1.map.get()
print d1.map.get()

Brian Jones wrote:
[color=blue]
> I'm sure the solution may be obvious, but this problem is driving me
> mad. The following is my code:
>[/color]

[Ken Rowlands]

Re: Class Variable Inheritance

On Wed, 08 Dec 2004 15:55:09 -0900, Brian Jones wrote:
[color=blue]
> I'm sure the solution may be obvious, but this problem is driving me
> mad. The following is my code:
>
> class a(object):
>
> mastervar = []
>
> def __init__(self):
> print 'called a'
>
> class b(a):
>
> def __init__(self):
> print 'called b'
> self.mapvar()
>
> def mapvar(self):
> self.mastervar. append(['b'])
>
> class c(b):
>
> def __init__(self):
> print 'called c'
> self.mapvar()
>
> def mapvar(self):
> super(c, self).mapvar()
> self.mastervar. append(['c'])
>
> if __name__ == '__main__':
>
> a1 = a()
> b1 = b()
> c1 = c()
> d1 = c() # Call C again
>
> print a1.mastervar
> print b1.mastervar
> print c1.mastervar
> print d1.mastervar
>
> What I don't understand is why mastervar gets modified by each _seperate
> instance_ of classes that happen to extend the base class 'a'.
> Shouldn't mastervar be contained within the scope of the inheriting
> classes? Why is it being treated like a global variable and being
> modified by the other instances?
>
>
> Thanks,
>
> Brian "bojo" Jones[/color]


Brian,

This is the first time I've responded to a post in this newsgroup so bear
with me.

First of all I believe that the place you declare mastervar is important.
As you have placed it above (or outside) any class methods/functions it
becomes a class variable. This gives it class scope and ALL instances of
that class (and sub-classes I think) will reference the SAME variable. If,
instead, you declare it inside the __init__ function of the class a object
it will be an instance variable and inheritable.

Secondly, in order to inherit methods and functions from base classes it
is important to remember to call the base class __init__ function in the
sub-class __init__ function.

If you do both these things, that is declare mastervar as 'self.mastervar
= []' inside class a.__init__ and call a.__init__ from b.__init__ (and
likewise for b in c) then you should get the expected result. I hope.

All I can say at this point is that I and not what you would call a
programmer, and I could be quite wrong in my explanation. However if this
doesn't help then there are many folk here who are better qualified to
give you the correct solution.


Regards

[Jeremy Bowers]

Re: Class Variable Inheritance

On Wed, 08 Dec 2004 16:49:34 -0900, Brian "bojo" Jones wrote:[color=blue]
> class a(object):
>
> def __init__(self):
> self.map = mapper()
> print 'called a'[/color]

What is the advantage of this over

def __init__(self):
self.map = []

?

[Steven Bethard]

Re: Class Variable Inheritance

Brian "bojo" Jones wrote:[color=blue]
> It became clear to me that mastervar inside of class a is a static
> variable, and is associated with all instances of classes that extend
> class a.[/color]

Yeah, that's basically what's happening. AFAICT, a variable declared at
class level is shared with all subclasses (and is available to all
instances unless hidden by an instance variable). You can simulate the
kind of behavior you want using descriptors:
[color=blue][color=green][color=darkred]
>>> import copy
>>> class subclass_copied (object):[/color][/color][/color]
.... def __init__(self, initial_value):
.... self.initial_va lue = initial_value
.... self.instances = {}
.... def __get__(self, instance, owner):
.... if owner not in self.instances:
.... self.instances[owner] = copy.copy(self. initial_value)
.... return self.instances[owner]
....[color=blue][color=green][color=darkred]
>>> class A(object):[/color][/color][/color]
.... x = subclass_copied ([])
....[color=blue][color=green][color=darkred]
>>> class B(A):[/color][/color][/color]
.... pass
....[color=blue][color=green][color=darkred]
>>> A.x.append(1)
>>> A().x.append(2)
>>> A.x[/color][/color][/color]
[1, 2][color=blue][color=green][color=darkred]
>>> B.x[/color][/color][/color]
[][color=blue][color=green][color=darkred]
>>> B.x.append(3)
>>> B().x.append(4)
>>> B.x[/color][/color][/color]
[3, 4][color=blue][color=green][color=darkred]
>>> A.x[/color][/color][/color]
[1, 2]

Basically, the subclass_copied descriptor returns a different object for
each class by keeping a type -> object dict. If you wanted to be
thorough with this, you would probably define a __set__ method too; see:

http://docs.python.org/ref/descriptors.html

Steve

Steve

[Craig Ringer]

Re: Class Variable Inheritance

On Thu, 2004-12-09 at 08:55, Brian Jones wrote:[color=blue]
> I'm sure the solution may be obvious, but this problem is driving me
> mad. The following is my code:
>
> class a(object):
>
> mastervar = []
>
> def __init__(self):
> print 'called a'
>
> class b(a):
>
> def __init__(self):
> print 'called b'
> self.mapvar()
>
> def mapvar(self):
> self.mastervar. append(['b'])
>
> class c(b):
>
> def __init__(self):
> print 'called c'
> self.mapvar()
>
> def mapvar(self):
> super(c, self).mapvar()
> self.mastervar. append(['c'])
>
> if __name__ == '__main__':
>
> a1 = a()
> b1 = b()
> c1 = c()
> d1 = c() # Call C again
>
> print a1.mastervar
> print b1.mastervar
> print c1.mastervar
> print d1.mastervar
>
> What I don't understand is why mastervar gets modified by each _seperate
> instance_ of classes that happen to extend the base class 'a'.
> Shouldn't mastervar be contained within the scope of the inheriting
> classes? Why is it being treated like a global variable and being
> modified by the other instances?[/color]

A variable declared in a class definition is shared by all instances of
the class. Python uses, essentially, a search path for names that goes
from most specific to least specific scope. In your example, a lookup in
c1 for mastervar will search for mastervar in:

c1.__dict__
c1.__class__.di ct (ie b.__dict__)
a.__dict__

Note that "class variables" are not _copied_, they're just looked up in
the class object if not found in the instance. Remember, the class
declaration is only executed _once_ - then __init__ is executed for each
new instance. If you want a per-instance copy of a variable, you need to
generate a new copy in the __init__ method of the instance (or the
parent class and call the parent's __init__ with super() ).

If you actually assigned c1.mastervar, rather than modifying the dict,
you would get the per-instance dictionary you expected.

I strongly recommend a read of the book "Learning Python" if you want to
really _understand_ this stuff (and they do a much better job explaining
it than I ever could in my overcomplicated and incoherent way).

I think the Python behaviour is counter-intuitive to C++ and Java
programmers, but makes good sense in the context of the way Python
classes, inheritance, and namespaces work. It's also extremely
consistent with the way namespaces and inheritance work in the rest of
Python - there's no special magic for objects and classes. A good rule
might be "If you want Java-style instance variables, create instances
variables in __init__ not in the class declaration."

This might help explain things - ore might just confuse even more:
[color=blue][color=green][color=darkred]
>>> class A(object):[/color][/color][/color]
.... ina = ["in A"]
....[color=blue][color=green][color=darkred]
>>> class B(A):[/color][/color][/color]
.... inb = ["in B"]
....[color=blue][color=green][color=darkred]
>>> class C(A):[/color][/color][/color]
.... ina = ["in C"]
.... inc = ["in C"]
....[color=blue][color=green][color=darkred]
>>> # examine the class dictionaries[/color][/color][/color]
....[color=blue][color=green][color=darkred]
>>> A.__dict__.keys ()[/color][/color][/color]
['ina', ...][color=blue][color=green][color=darkred]
>>> B.__dict__.keys ()[/color][/color][/color]
['inb', ...][color=blue][color=green][color=darkred]
>>> C.__dict__.keys ()[/color][/color][/color]
['ina', 'inc', ...][color=blue][color=green][color=darkred]
>>> # Now look up some variables to demonstrate the namespace[/color][/color][/color]
.... # search in class inheritance
....[color=blue][color=green][color=darkred]
>>> A.ina[/color][/color][/color]
['in A'][color=blue][color=green][color=darkred]
>>> B.ina[/color][/color][/color]
['in A'][color=blue][color=green][color=darkred]
>>> C.ina # remember, we redefined this in C[/color][/color][/color]
['in C'][color=blue][color=green][color=darkred]
>>> B.inb[/color][/color][/color]
['in B'][color=blue][color=green][color=darkred]
>>> C.inc[/color][/color][/color]
['in C'][color=blue][color=green][color=darkred]
>>> # This should help explain things[/color][/color][/color]
....[color=blue][color=green][color=darkred]
>>> B.ina is A.ina # True, because B.ina just looks up A.ina[/color][/color][/color]
True[color=blue][color=green][color=darkred]
>>> C.ina is A.ina # False, because C.ina is found first[/color][/color][/color]
False[color=blue][color=green][color=darkred]
>>> # Now modify B.ina. Because asking for B.ina searches B, then A,[/color][/color][/color]
.... # for ina, we'll actually end up modifying A.ina
....[color=blue][color=green][color=darkred]
>>> B.ina.append("b lah")
>>> A.ina[/color][/color][/color]
['in A', 'blah'][color=blue][color=green][color=darkred]
>>> B.ina[/color][/color][/color]
['in A', 'blah'][color=blue][color=green][color=darkred]
>>> # but if we do the same to C.ina, which is redefined in C,[/color][/color][/color]
.... # a.ina won't be modified
....[color=blue][color=green][color=darkred]
>>> C.ina.append("c hange")
>>> A.ina[/color][/color][/color]
['in A', 'blah'][color=blue][color=green][color=darkred]
>>> C.ina[/color][/color][/color]
['in C', 'change'][color=blue][color=green][color=darkred]
>>> # Now we're going to assign to B.ina, rebinding the name,[/color][/color][/color]
.... # instead of just modifying the existing mutable object.
....[color=blue][color=green][color=darkred]
>>> B.ina = "fred"
>>> B.ina[/color][/color][/color]
"fred"[color=blue][color=green][color=darkred]
>>> B.__dict__.keys ()[/color][/color][/color]
['inb', 'ina'][color=blue][color=green][color=darkred]
>>> # Note that there's now a new value for ina in B's dictionary. It[/color][/color][/color]
.... # is found by the search BEFORE looking for A.ina, and used. A.ina
.... # is not modified.
....[color=blue][color=green][color=darkred]
>>> A.ina[/color][/color][/color]
['in A', 'blah']


What you can see happening here is the combination of a couple of
principles:
- Name lookups in classes happen as a search through the class
dictionary then all its parent classes' dictionaries for a name.
- Name assignments to a class are made directly in its dictionary.
- A modification of a mutable value is a lookup of a name followed
by the modification of an object, not an assignment to a name.

The principle is pretty similar for instances and classes - a variable
defined in a class is a single object shared between all instances of a
class, and if mutable can be modified by all of them. Example:
[color=blue][color=green][color=darkred]
>>> class D(object):[/color][/color][/color]
.... cvar = []
.... def __init__(self, name):
.... self.cvar.appen d(name)
.... print "I am ", name
....[color=blue][color=green][color=darkred]
>>> a = D("fred")[/color][/color][/color]
I am fred[color=blue][color=green][color=darkred]
>>> b = D("jones")[/color][/color][/color]
I am jones[color=blue][color=green][color=darkred]
>>> D.cvar[/color][/color][/color]
['fred', 'jones'][color=blue][color=green][color=darkred]
>>> a.cvar is D.cvar[/color][/color][/color]
True

If, on the other hand, I assign to that name - rather than modifying a
mutable object:
[color=blue][color=green][color=darkred]
>>> class E(object):[/color][/color][/color]
.... name = "fred"
.... def __init__(self, name):
.... self.name = name
....[color=blue][color=green][color=darkred]
>>> a = E("smith")
>>> b = E("anna")
>>> E.name[/color][/color][/color]
'fred'[color=blue][color=green][color=darkred]
>>> a.name[/color][/color][/color]
'smith'[color=blue][color=green][color=darkred]
>>> b.name[/color][/color][/color]
'anna'[color=blue][color=green][color=darkred]
>>> E.name is a.name[/color][/color][/color]
False[color=blue][color=green][color=darkred]
>>> a.__dict__.keys ()[/color][/color][/color]
['name'][color=blue][color=green][color=darkred]
>>> b.__dict__.keys ()[/color][/color][/color]
['name'][color=blue][color=green][color=darkred]
>>> E.__dict__.keys ()
>>> E.__dict__.keys ()[/color][/color][/color]
['name', ...]

a new entry is inserted into each instances dictionary. The class
variable is still there, and unmodified, but the name search finds the
copy each instance has in its dict before the class one.
[color=blue][color=green][color=darkred]
>>> a.__class__.nam e[/color][/color][/color]
'fred'[color=blue][color=green][color=darkred]
>>> a.__class__.nam e = "Albert"
>>> a.__class__.nam e[/color][/color][/color]
'fred'[color=blue][color=green][color=darkred]
>>> a.name[/color][/color][/color]
'smith'

See?

The handling of names on instances and subclasses is a lot like the way
names are handled in scopes in functions. Lookups scan upward from most
specific to least specific scope, ending at the global module scope,
while assignments bind a name into the local scope. Again, modifying a
mutable global works the same way as modifying a mutable member of a
class object.

I know it sounds darn complicated, but once you "get it" about namespace
searches, the difference between modifying an object and binding an
name, etc it's all very clean and nice and easy. I love the way
inheritance and name lookups work in Python - it's all 100% consistent.

The best way to learn it is play with it, though reading Learning Python
is strongly recommended IMHO.

You can use this behaviour to your _strong_ benefit, with
instance-caching classes (requires overriding __new__), classes that
automatically register and keep track of instances, and instances that
can all inherit a change made to the class object even after they're
instantiated. I've found instance caching, in particular, to be just
magic when performing lots of data processing.

--
Craig Ringer

[Scott David Daniels]

Re: Class Variable Inheritance

Brian Jones wrote:[color=blue]
> class a(object):
> mastervar = []
> def __init__(self):
> print 'called a'
>
> class b(a):
> def __init__(self):
> print 'called b'
> self.mapvar()
> def mapvar(self):
> self.mastervar. append(['b'])
>
> class c(b):[/color]
mastervar = [] # Adding this should make things clearer[color=blue]
> def __init__(self):
> print 'called c'
> self.mapvar()
> def mapvar(self):
> super(c, self).mapvar()
> self.mastervar. append(['c'])
>
> if __name__ == '__main__':
> a1 = a()
> b1 = b()
> c1 = c()
> d1 = c() # Call C again[/color]
for object in a1, b1, c1, d1:
print id(object.maste rvar), object.masterva r
[color=blue]
> What I don't understand is why mastervar gets modified by each _seperate
> instance_ of classes that happen to extend the base class 'a'. Shouldn't
> mastervar be contained within the scope of the inheriting classes? Why
> is it being treated like a global variable and being modified by the
> other instances?[/color]
By over-riding mastervar in class c, I hope I've shown that a class
variable is shared by all of its instances, but can be over-ridden by
a subclass's class variable.

--Scott David Daniels
Scott.Daniels@A cm.Org