classmethods, class variables and subclassing

[Andrew Jaffe]

Hi,

I have a class with various class-level variables which are used to
store global state information for all instances of a class. These are
set by a classmethod as in the following (in reality the setcvar method
is more complicated than this!):

class sup(object):
cvar1 = None
cvar2 = None

@classmethod
def setcvar1(cls, val):
cls.cvar1 = val

@classmethod
def setcvar2(cls, val):
cls.cvar2 = val

@classmethod
def printcvars(cls) :
print cls.cvar1, cls.cvar2


I can then call setcvar on either instances of the class or the class
itself.

Now, the problem comes when I want to subclass this class. If I
override the setcvar1 method to do some new things special to this
class, and then call the sup.setcvar1() method, it all works fine:

class sub(sup):
cvar1a = None

@classmethod
def setcvar1(cls, val, vala):
cls.cvar1a = vala
sup.setcvar1(va l)

@classmethod
def printcvars(cls) :
print cls.cvar1a
sup.printcvars( )

This works fine, and sets cvar and cvar2 for both classes.

However, if I *don't* override the setcvar2 method, but I call
sub.setcvar2(va l) directly, then only sub.cvar2 gets set; it is no
longer identical to sup.cvar1!

In particular,
sub.setcvar1(1, 10)
sub.setcvar2(2)
sub.printcvars( )
prints
10
1 None

i.e. sub.cvar1, sub.cvar1a, sub.cvar2= 1 10 2
but sup.cvar1, cvar2= 1 None

since sup.cvar2 has never been set, and this is what sup.printcvars( )
looks for.

This behavior is "expected", but is it desirable?

For my application, at least, I think the problem really comes in the
printcvars method: is there any way to call the overridden
sup.printcvars( ) but with, effectively, cls=sub?

Thanks for reading this far!

Andrew


Andrew




This seems like a bug

Is this expected behavior, or a bug (or both -- it is expected but
probably not what is wanted!)?

[Steve Holden]

Re: classmethods, class variables and subclassing

Andrew Jaffe wrote:[color=blue]
> Hi,
>
> I have a class with various class-level variables which are used to
> store global state information for all instances of a class. These are
> set by a classmethod as in the following (in reality the setcvar method
> is more complicated than this!):
>
> class sup(object):
> cvar1 = None
> cvar2 = None
>
> @classmethod
> def setcvar1(cls, val):
> cls.cvar1 = val
>
> @classmethod
> def setcvar2(cls, val):
> cls.cvar2 = val
>
> @classmethod
> def printcvars(cls) :
> print cls.cvar1, cls.cvar2
>
>
> I can then call setcvar on either instances of the class or the class
> itself.
>
> Now, the problem comes when I want to subclass this class. If I
> override the setcvar1 method to do some new things special to this
> class, and then call the sup.setcvar1() method, it all works fine:
>
> class sub(sup):
> cvar1a = None
>
> @classmethod
> def setcvar1(cls, val, vala):
> cls.cvar1a = vala
> sup.setcvar1(va l)
>
> @classmethod
> def printcvars(cls) :
> print cls.cvar1a
> sup.printcvars( )
>
> This works fine, and sets cvar and cvar2 for both classes.
>
> However, if I *don't* override the setcvar2 method, but I call
> sub.setcvar2(va l) directly, then only sub.cvar2 gets set; it is no
> longer identical to sup.cvar1!
>
> In particular,
> sub.setcvar1(1, 10)
> sub.setcvar2(2)
> sub.printcvars( )
> prints
> 10
> 1 None
>
> i.e. sub.cvar1, sub.cvar1a, sub.cvar2= 1 10 2
> but sup.cvar1, cvar2= 1 None
>
> since sup.cvar2 has never been set, and this is what sup.printcvars( )
> looks for.
>
> This behavior is "expected", but is it desirable?
>
> For my application, at least, I think the problem really comes in the
> printcvars method: is there any way to call the overridden
> sup.printcvars( ) but with, effectively, cls=sub?
>
> Thanks for reading this far!
>
> Andrew
>
>
> Andrew
>
>
>
>
> This seems like a bug
>
> Is this expected behavior, or a bug (or both -- it is expected but
> probably not what is wanted!)?[/color]

You are experiencing this problem because you are using hard-wired class
names. Try using (for example) self.__class__. That way, even if your
method is inheroted by a subclass it will use the class of the object it
finds itself a method of. No need to use classmethods.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Andrew Jaffe]

Re: classmethods, class variables and subclassing

> Andrew Jaffe wrote:[color=blue]
>[color=green]
>> Hi,
>>
>> I have a class with various class-level variables which are used to
>> store global state information for all instances of a class. These are
>> set by a classmethod as in the following
>>
>> class sup(object):
>> cvar1 = None
>> cvar2 = None
>>
>> @classmethod
>> def setcvar1(cls, val):
>> cls.cvar1 = val
>>
>> @classmethod
>> def setcvar2(cls, val):
>> cls.cvar2 = val
>>
>> @classmethod
>> def printcvars(cls) :
>> print cls.cvar1, cls.cvar2
>>
>> Now, the problem comes when I want to subclass this class. If I
>> override the setcvar1 method to do some new things special to this
>> class, and then call the sup.setcvar1() method, it all works fine:
>>
>> class sub(sup):
>> cvar1a = None
>>
>> @classmethod
>> def setcvar1(cls, val, vala):
>> cls.cvar1a = vala
>> sup.setcvar1(va l)
>>
>> @classmethod
>> def printcvars(cls) :
>> print cls.cvar1a
>> sup.printcvars( )
>>
>> This works fine, and sets cvar and cvar2 for both classes.
>>
>> However, if I *don't* override the setcvar2 method, but I call
>> sub.setcvar2(va l) directly, then only sub.cvar2 gets set; it is no
>> longer identical to sup.cvar1!
>>
>> In particular,
>> sub.setcvar1(1, 10)
>> sub.setcvar2(2)
>> sub.printcvars( )
>> prints
>> 10
>> 1 None
>>
>> i.e. sub.cvar1, sub.cvar1a, sub.cvar2= 1 10 2
>> but sup.cvar1, cvar2= 1 None
>>
>> This behavior is "expected", but is it desirable?
>>[/color]
>
> You are experiencing this problem because you are using hard-wired class
> names. Try using (for example) self.__class__. That way, even if your
> method is inheroted by a subclass it will use the class of the object it
> finds itself a method of. No need to use classmethods.[/color]

The problem is that I actually do want to call these methods on the
class itself, before I've made any instances.

A

[Steve Holden]

Re: classmethods, class variables and subclassing

Andrew Jaffe wrote:[color=blue][color=green]
>>Andrew Jaffe wrote:
>>
>>[color=darkred]
>>>Hi,
>>>
>>>I have a class with various class-level variables which are used to
>>>store global state information for all instances of a class. These are
>>>set by a classmethod as in the following
>>>
>>>class sup(object):
>>> cvar1 = None
>>> cvar2 = None
>>>
>>> @classmethod
>>> def setcvar1(cls, val):
>>> cls.cvar1 = val
>>>
>>> @classmethod
>>> def setcvar2(cls, val):
>>> cls.cvar2 = val
>>>
>>> @classmethod
>>> def printcvars(cls) :
>>> print cls.cvar1, cls.cvar2
>>>
>>>Now, the problem comes when I want to subclass this class. If I
>>>override the setcvar1 method to do some new things special to this
>>>class, and then call the sup.setcvar1() method, it all works fine:
>>>
>>>class sub(sup):
>>> cvar1a = None
>>>
>>> @classmethod
>>> def setcvar1(cls, val, vala):
>>> cls.cvar1a = vala
>>> sup.setcvar1(va l)
>>>
>>> @classmethod
>>> def printcvars(cls) :
>>> print cls.cvar1a
>>> sup.printcvars( )
>>>
>>>This works fine, and sets cvar and cvar2 for both classes.
>>>
>>>However, if I *don't* override the setcvar2 method, but I call
>>>sub.setcvar2 (val) directly, then only sub.cvar2 gets set; it is no
>>>longer identical to sup.cvar1!
>>>
>>>In particular,
>>> sub.setcvar1(1, 10)
>>> sub.setcvar2(2)
>>> sub.printcvars( )
>>>prints
>>> 10
>>> 1 None
>>>
>>>i.e. sub.cvar1, sub.cvar1a, sub.cvar2= 1 10 2
>>>but sup.cvar1, cvar2= 1 None
>>>
>>>This behavior is "expected", but is it desirable?
>>>[/color]
>>
>>You are experiencing this problem because you are using hard-wired class
>>names. Try using (for example) self.__class__. That way, even if your
>>method is inheroted by a subclass it will use the class of the object it
>>finds itself a method of. No need to use classmethods.[/color]
>
>
> The problem is that I actually do want to call these methods on the
> class itself, before I've made any instances.
>[/color]
I see. I think. So what you are saying is that when you call
sup.printcvars( ) from inside a sub method you want it to see the
namespace of the sub class not the sup?

Since you have set all this up carefully you must have a use case, but
it seems a little contorted (to me). Basically you appear to want the
classes to behave statically the way that instances do dynamically?

Not sure I can help you here.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Steve Holden]

Re: classmethods, class variables and subclassing

Andrew Jaffe wrote:[color=blue][color=green]
>>Andrew Jaffe wrote:[/color][/color]
[...][color=blue]
>
> The problem is that I actually do want to call these methods on the
> class itself, before I've made any instances.
>[/color]
Except you could use staticmethods with an explicit class argument ...

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Andrew Jaffe]

Re: classmethods, class variables and subclassing

Steve Holden wrote:[color=blue]
> Andrew Jaffe wrote:
>[color=green]
>>The problem is that I actually do want to call these methods on the
>>class itself, before I've made any instances.
>>[/color]
> Except you could use staticmethods with an explicit class argument ...[/color]

Steve,

Yep, that would work! Thanks.

But it does seem like a bit of a kludge: classmethods seems to be almost
exactly what you 'ought' to use here (i.e., I really do want to apply
these methods to the class as an object in its own right).

A

[Steve Holden]

Re: classmethods, class variables and subclassing

Andrew Jaffe wrote:[color=blue]
> Steve Holden wrote:
>[color=green]
>>Andrew Jaffe wrote:
>>
>>[color=darkred]
>>>The problem is that I actually do want to call these methods on the
>>>class itself, before I've made any instances.
>>>[/color]
>>
>>Except you could use staticmethods with an explicit class argument ...[/color]
>
>
> Steve,
>
> Yep, that would work! Thanks.
>
> But it does seem like a bit of a kludge: classmethods seems to be almost
> exactly what you 'ought' to use here (i.e., I really do want to apply
> these methods to the class as an object in its own right).
>[/color]
What's the use case?

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Andrew Jaffe]

Re: classmethods, class variables and subclassing

Steve Holden wrote:[color=blue]
> Andrew Jaffe wrote:[color=green]
>> Steve Holden wrote:[color=darkred]
>>> Andrew Jaffe wrote:
>>>
>>>> The problem is that I actually do want to call these methods on the
>>>> class itself, before I've made any instances.
>>>>
>>> Except you could use staticmethods with an explicit class argument ...[/color]
>>
>> Yep, that would work! Thanks.
>>
>> But it does seem like a bit of a kludge: classmethods seems to be
>> almost exactly what you 'ought' to use here (i.e., I really do want to
>> apply these methods to the class as an object in its own right).
>>[/color]
> What's the use case?[/color]

Fair question. I hope the following isn't too technical.

I have a class which describes the model for fitting some data,
encapsulating among other things a bunch of parameters whose values I'd
like to determine for a given dataset. The base class is a simple model,
the derived class a slightly more complicated one with an extra parameter.

At present, I instantiate the class with a particular set of values for
the parameters.

One of the methods in this class is called 'prior', which returns the
prior probability for the instance's paramters.

However, it turns out there are some 'meta-parameters' which don't
change between instances, in particular the allowed limits on the
parameters, beyond which the prior should return 0. Currently these are
stored as class variables -- so both the base and derived class want to
be able to act as if these class variables are 'native' to their own
class -- since in fact the base/derived relationship is in this case
actually something of an implementation detail. (Currently I've solved
the problem by explicitly using the base class methods for setting the
class variables.)

Does this make sense?

Andrew

[Steve Holden]

Re: classmethods, class variables and subclassing

Andrew Jaffe wrote:[color=blue]
> Steve Holden wrote:
>[color=green]
>>Andrew Jaffe wrote:
>>[color=darkred]
>>>Steve Holden wrote:
>>>
>>>>Andrew Jaffe wrote:
>>>>
>>>>
>>>>>The problem is that I actually do want to call these methods on the
>>>>>class itself, before I've made any instances.
>>>>>
>>>>
>>>>Except you could use staticmethods with an explicit class argument ...
>>>
>>>Yep, that would work! Thanks.
>>>
>>>But it does seem like a bit of a kludge: classmethods seems to be
>>>almost exactly what you 'ought' to use here (i.e., I really do want to
>>>apply these methods to the class as an object in its own right).
>>>[/color]
>>
>>What's the use case?[/color]
>
>
> Fair question. I hope the following isn't too technical.
>
> I have a class which describes the model for fitting some data,
> encapsulating among other things a bunch of parameters whose values I'd
> like to determine for a given dataset. The base class is a simple model,
> the derived class a slightly more complicated one with an extra parameter.
>
> At present, I instantiate the class with a particular set of values for
> the parameters.
>
> One of the methods in this class is called 'prior', which returns the
> prior probability for the instance's paramters.
>
> However, it turns out there are some 'meta-parameters' which don't
> change between instances, in particular the allowed limits on the
> parameters, beyond which the prior should return 0. Currently these are
> stored as class variables -- so both the base and derived class want to
> be able to act as if these class variables are 'native' to their own
> class -- since in fact the base/derived relationship is in this case
> actually something of an implementation detail. (Currently I've solved
> the problem by explicitly using the base class methods for setting the
> class variables.)
>
> Does this make sense?
>[/color]

I think so. It's not normal adive, but it sounds like a metaclass might
be what you need here.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Steve Holden]

Re: classmethods, class variables and subclassing

Steve Holden wrote:
[...][color=blue]
>
> I think so. It's not normal adive, but it sounds like a metaclass might
> be what you need here.
>[/color]
^adive^advice^

spell-me-own-name-wrong-next-ly y'rs - evest
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Steven Bethard]

Re: classmethods, class variables and subclassing

Andrew Jaffe wrote:[color=blue]
> Hi,
>
> I have a class with various class-level variables which are used to
> store global state information for all instances of a class. These are
> set by a classmethod as in the following (in reality the setcvar method
> is more complicated than this!):
>
> class sup(object):
> cvar1 = None
> cvar2 = None
>
> @classmethod
> def setcvar1(cls, val):
> cls.cvar1 = val
>
> @classmethod
> def setcvar2(cls, val):
> cls.cvar2 = val
>
> @classmethod
> def printcvars(cls) :
> print cls.cvar1, cls.cvar2
>
>
> I can then call setcvar on either instances of the class or the class
> itself.
>
> Now, the problem comes when I want to subclass this class. If I override
> the setcvar1 method to do some new things special to this class, and
> then call the sup.setcvar1() method, it all works fine:
>
> class sub(sup):
> cvar1a = None
>
> @classmethod
> def setcvar1(cls, val, vala):
> cls.cvar1a = vala
> sup.setcvar1(va l)
>
> @classmethod
> def printcvars(cls) :
> print cls.cvar1a
> sup.printcvars( )
>
> This works fine, and sets cvar and cvar2 for both classes.
>
> However, if I *don't* override the setcvar2 method, but I call
> sub.setcvar2(va l) directly, then only sub.cvar2 gets set; it is no
> longer identical to sup.cvar1!
>
> In particular,
> sub.setcvar1(1, 10)
> sub.setcvar2(2)
> sub.printcvars( )
> prints
> 10
> 1 None
>
> i.e. sub.cvar1, sub.cvar1a, sub.cvar2= 1 10 2
> but sup.cvar1, cvar2= 1 None[/color]

I'm not sure if I understand your goal here, but you can get different
behavior using super().

py> class sup(object):
.... cvar1 = None
.... cvar2 = None
.... @classmethod
.... def setcvar1(cls, val):
.... cls.cvar1 = val
.... @classmethod
.... def setcvar2(cls, val):
.... cls.cvar2 = val
.... @classmethod
.... def printcvars(cls) :
.... print cls.cvar1, cls.cvar2
....
py> class sub(sup):
.... cvar1a = None
.... @classmethod
.... def setcvar1(cls, val, vala):
.... cls.cvar1a = vala
.... super(sub, cls).setcvar1(v al)
.... @classmethod
.... def printcvars(cls) :
.... print cls.cvar1a
.... super(sub, cls).printcvars ()
....
py> sub.setcvar1(1, 10); sub.setcvar2(2) ; sub.printcvars( )
10
1 2
py> sup.printcvars( )
None None

I'm not sure what you want sup.printcvars( ) to print afterwards. If you
want it to print out "1 2" instead of "None None", then what you're
trying to do is to set every cvar in every superclass. You'll need to
be explicit about this, perhaps something like:

py> class sup(object):
.... cvar1 = None
.... cvar2 = None
.... @classmethod
.... def setcvar1(cls, val):
.... for cls in cls.mro()[:-1]: # search through superclasses
.... cls.cvar1 = val
.... @classmethod
.... def setcvar2(cls, val):
.... for cls in cls.mro()[:-1]: # search through superclasses
.... cls.cvar2 = val
.... @classmethod
.... def printcvars(cls) :
.... print cls.cvar1, cls.cvar2
....
py> class sub(sup):
.... cvar1a = None
.... @classmethod
.... def setcvar1(cls, val, vala):
.... for cls in cls.mro()[:-2]: # search through superclasses
.... cls.cvar1a = vala
.... super(sub, cls).setcvar1(v al)
.... @classmethod
.... def printcvars(cls) :
.... print cls.cvar1a
.... super(sub, cls).printcvars ()
....
py> sub.setcvar1(1, 10); sub.setcvar2(2) ; sub.printcvars( )
10
1 2
py> sup.printcvars( )
1 2

That is, if you want the cvar set on every superclass, you need an
assignment statement for every superclass. There's probably a way to
factor out the for-loop so you don't have to write it every time, but I
haven't thought about it too much yet. Perhaps an appropriate
descriptor in the metaclass...

STeVe

[Ron Adam]

Re: classmethods, class variables and subclassing

Andrew Jaffe wrote:
[color=blue]
> Hi,
>
> I have a class with various class-level variables which are used to
> store global state information for all instances of a class. These are
> set by a classmethod as in the following (in reality the setcvar method
> is more complicated than this!):
>
> class sup(object):
> cvar1 = None
> cvar2 = None
>
> @classmethod
> def setcvar1(cls, val):
> cls.cvar1 = val
>
> @classmethod
> def setcvar2(cls, val):
> cls.cvar2 = val
>
> @classmethod
> def printcvars(cls) :
> print cls.cvar1, cls.cvar2[/color]

How about just using a mutable list?

class sup(object):
states = [None,None] # [s1, s2, ...]
def set_s1(self, val):
self.states[0] = val
def get_s1(self):
return self.states[0]
def set_s2(self, val):
self.states[1] = val
def get_s2(self):
return self.states[1]

It keeps the states because the list isn't ever reassigned after it's
created, so the values in it can change and all instances and subclasses
can see the changed values.

Cheers,
Ron

[Andrew Jaffe]

Re: classmethods, class variables and subclassing

Steven Bethard wrote:[color=blue]
> Andrew Jaffe wrote:
>
> I'm not sure if I understand your goal here, but you can get different
> behavior using super().
>
> py> class sup(object):
> ... cvar1 = None
> ... cvar2 = None
> ... @classmethod
> ... def setcvar1(cls, val):
> ... cls.cvar1 = val
> ... @classmethod
> ... def setcvar2(cls, val):
> ... cls.cvar2 = val
> ... @classmethod
> ... def printcvars(cls) :
> ... print cls.cvar1, cls.cvar2
> ...
> py> class sub(sup):
> ... cvar1a = None
> ... @classmethod
> ... def setcvar1(cls, val, vala):
> ... cls.cvar1a = vala
> ... super(sub, cls).setcvar1(v al)
> ... @classmethod
> ... def printcvars(cls) :
> ... print cls.cvar1a
> ... super(sub, cls).printcvars ()
> ...
> py> sub.setcvar1(1, 10); sub.setcvar2(2) ; sub.printcvars( )
> 10
> 1 2
> py> sup.printcvars( )
> None None[/color]

Aha! This is the behavior I want -- the variables get set correctly when
the methods are called on the subclass. I thought super() might be the
right idea, but I didn't realize you could call it as super(sub, cls)
rather than with an instance in the second slot. I don't think my use
case ever needs the superclass to have the variables accessible as in
your next ideas, not reproduced here.

Thanks!

Andrew