Late initialization using getattribute

**bukzor** · Sep 3 '08, 09:55 PM

Re: Late initialization using __getattribute_ _

On Sep 3, 12:19 pm, Bruno Desthuilliers
<bdesth.quelque ch...@free.quel quepart.frwrote :

bukzor a écrit :
>

I want to make a MixIn class that waits to initialize its super-
classes until an attribute of the object is accessed. Not generally
useful, but desirable in my case. I've written this, and it works, but
would like to take any suggestions you guys have.

>
You shouldn't mess with __getattribute_ _ unless you really know what
you're doing and are ok to suffer the constant performance hit you'll
get. Remember that __getattribute_ _ actually *is* the implementation of
attribute lookup rules, and is called on each and every attribute
lookup. Your below snippet would be much better using __getattr__ (which
is only called as a last resort).
>

I've commented out
the "delattr" call because it throws an AttributeError (although I
don't know why).

>
__getattribute_ _ belongs to the class, not to the instance. delattr()
only removes instance attributes. You'd have to remove __getattribute_ _
from the LateInitMixIn class object itself, but then it would break the
whole thing.
>
>
>

class LateInitMixIn(o bject):
def __init__(self):
print "LateInit initialization"
self.inited = False
def __getattribute_ _(self, attr):
print "Doing __getattribute_ _"
getattr = lambda attr:object.__g etattribute__(s elf, attr)
if not getattr("inited "):
super(LateInitM ixIn, self).__init__( )
setattr(self, "inited", True)
#delattr(self, "__getattribute __")
return getattr(attr)

>
Here's another possible implementation (which doesn't solve all
problems, cf below) using __getattr__:
>
class LateInitMixin(o bject):
def __init__(self):
print "not yet"
self.__initiali zed = False
>
def __getattr__(sel f, name):
if self.__initiali zed:
raise AttributeError(
"object %s has no attribute '%s'" % (type(self), name)
)
super(LateInitM ixin, self).__init__( )
self.__initiali zed = True
return getattr(self, name)
>
class Base(object):
def __init__(self):
print "yet"
self.base = True
>
class LateInit(LateIn itMixin, Base):
pass
>
def main():
print "shouldn't init"
S = LateInit()
print "should init"
print S.base
>
if __name__=="__ma in__":
main()
>
Ok, now, the other problem : what if Base.__init__ expects args ?

Thanks for the reply. Just to see it not work, I tried to remove
__getattribute_ _ from LateInitMixIn, but couldn't get it to work.

My Base class is a C class (_mysql.connect ion from MySQLdb) that
sometimes segfaults if you try to use it before it's fully
initialized, so unfortunately I think I need to use __getattribute_ _
to do this. I'm doing all this just to make the connection not
actually connect until used.

**bukzor** · Sep 3 '08, 10:35 PM

Re: Late initialization using __getattribute_ _

On Sep 3, 1:02 pm, Bruno Desthuilliers
<bdesth.quelque ch...@free.quel quepart.frwrote :

bukzor a écrit :
(snip)
>

Thanks for the reply. Just to see it not work, I tried to remove
__getattribute_ _ from LateInitMixIn, but couldn't get it to work.

>
??? Sorry, I don't get what you mean...

Since you said __getattribute_ _ is an attribute of the class, I tried
to run (something to the effect of) delattr(self.__ class__,
"__getattribute __"), but it threw an AttributeError.

>

My Base class is a C class (_mysql.connect ion from MySQLdb) that
sometimes segfaults if you try to use it before it's fully
initialized,

>
>
... I have used MySQLdb for years on more than a dozen linux
distribs, and never had such a problem. Is this a known bug ? Or is
there something wrong with your setup ?

I'm trying to optimize my number of connections by not fully
initializing (read: not connecting) my connection until it's used in
some way. Of course the maintainer didn't envision this (mis)use, so
the object sometimes throws bad errors until it's fully initialized.

Of course the *correct* way to do this is to be more careful about
when I create connections, but I think I should be able to get this to
work, and it (would be) much easier to do it The Right Way once it
works.

so unfortunately I think I need to use __getattribute_ _
to do this. I'm doing all this just to make the connection not
actually connect until used.

>
I may be dumb, but I don't get how this is supposed to solve your
problem. But anyway : there's a known design pattern for what you're
trying to do, that doesn't require mixins nor messing with
__getattribute_ _ (which, I repeat, is more often than not something you
*don't* want to do). The name of the design pattern is "proxy". I
strongly suggest that you 1/ try to cure the real problem instead of
hacking around and 2/ read about the proxy design pattern.
>
My 2 cents...

I like the idea of mix-ins, but can't figure out how to make a proxy
work that way. For a long time I had a proxy class that added five or
six features on top of the MySQLdb package, but it wasn't configurable
enough, and I'm working on splitting each feature into its own MixIn
class.

As an aside, this is working for me pretty well. The "reconnect"
method (inheritied from a "Reconnecta ble" mixin) uses several of the
object's attributes, so I need to set _inited beforehand so that I
don't get into an infinite __getattribute_ _ loop. What I'd *really*
like to do is remove __getattribute_ _ from the object at that point.
def __getattribute_ _(self, attr):
"connect if it would otherwise cause an error."
getattr = lambda attr:object.__g etattribute__(s elf, attr)

if not getattr("_inite d"):
print "connecting ."
setattr(self, "_inited", True)
getattr("reconn ect")()
return getattr(attr)

Thanks for your help,
--Buck

**Michele Simionato** · Sep 4 '08, 03:35 AM

Re: Late initialization using __getattribute_ _

On Sep 4, 12:26 am, bukzor <workithar...@g mail.comwrote:

I'm trying to optimize my number of connections by not fully
initializing (read: not connecting) my connection until it's used in
some way.

I had the same use case and I solved with a simple property. Here is
the code
I have for pymssql:

@property
def conn(self):
if self._conn is None:
self._conn = _mssql.connect( self.host, self.user,self. passwd)

self._conn.sele ct_db(self.dbna me)
return self._conn

The connection is really instantiate only when you call self.conn to
perform the query, not when you instantiate the class.

**Bruno Desthuilliers** · Sep 4 '08, 08:25 AM

Re: Late initialization using __getattribute_ _

bukzor a écrit :

On Sep 3, 1:02 pm, Bruno Desthuilliers
<bdesth.quelque ch...@free.quel quepart.frwrote :

>bukzor a écrit :
>(snip)
>>

>>Thanks for the reply. Just to see it not work, I tried to remove
>>__getattribut e__ from LateInitMixIn, but couldn't get it to work.

>??? Sorry, I don't get what you mean...

>
Since you said __getattribute_ _ is an attribute of the class, I tried
to run (something to the effect of) delattr(self.__ class__,
"__getattribute __"),

Ah, ok. Definitively *not* a thing to do...

but it threw an AttributeError.

You'd have to call it on the mixin class itself, not on a subclass.

>

>>My Base class is a C class (_mysql.connect ion from MySQLdb) that
>>sometimes segfaults if you try to use it before it's fully
>>initialized ,

>>
>... I have used MySQLdb for years on more than a dozen linux
>distribs, and never had such a problem. Is this a known bug ? Or is
>there something wrong with your setup ?

>
I'm trying to optimize my number of connections

Connection pooling anyone ? IIRC, this is something that already exists
in quite a couple libs / frameworks. Is there a reason you try to roll
your own ?

by not fully
initializing (read: not connecting) my connection until it's used in
some way. Of course the maintainer didn't envision this (mis)use, so
the object sometimes throws bad errors until it's fully initialized.

Ok.

Of course the *correct* way to do this is to be more careful about
when I create connections, but I think I should be able to get this to
work, and it (would be) much easier to do it The Right Way once it
works.

Please pardon me for repeating the same thing over and over, but messing
with __getattribute_ _ is certainly not the way to go.

>>so unfortunately I think I need to use __getattribute_ _
>>to do this. I'm doing all this just to make the connection not
>>actually connect until used.

>I may be dumb, but I don't get how this is supposed to solve your
>problem. But anyway : there's a known design pattern for what you're
>trying to do, that doesn't require mixins nor messing with
>__getattribute __ (which, I repeat, is more often than not something you
>*don't* want to do). The name of the design pattern is "proxy". I
>strongly suggest that you 1/ try to cure the real problem instead of
>hacking around and 2/ read about the proxy design pattern.
>>
>My 2 cents...

>
I like the idea of mix-ins, but can't figure out how to make a proxy
work that way.

You mean, "how to use a proxy for lazy initialization" ? Heck, that's
the exact use case in the GoF.

For a long time I had a proxy class that added five or
six features on top of the MySQLdb package, but it wasn't configurable
enough, and I'm working on splitting each feature into its own MixIn
class.
>
>
As an aside, this is working for me pretty well. The "reconnect"
method (inheritied from a "Reconnecta ble" mixin) uses several of the
object's attributes, so I need to set _inited beforehand so that I
don't get into an infinite __getattribute_ _ loop. What I'd *really*
like to do is remove __getattribute_ _ from the object at that point.

You can't. Or, more exactly, all you can do is remove __getattribute_ _
from the mixin class - but then the mixin class won't work anymore. I
don't mean to be condescendant, but it looks like you don't have a clear
understanding of Python's object model here - else you wouldn't even
consider doing such a thing. FWIW, I posted a solution based on the
__getattr__ hook, which did work - at least for the "specs" implied by
your code snippet.

**bukzor** · Sep 4 '08, 04:45 PM

Re: Late initialization using __getattribute_ _

>so unfortunately I think I need to use __getattribute_ _
>to do this. I'm doing all this just to make the connection not
>actually connect until used.
I may be dumb, but I don't get how this is supposed to solve your
problem. But anyway : there's a known design pattern for what you're
trying to do, that doesn't require mixins nor messing with
__getattribute_ _ (which, I repeat, is more often than not something you
*don't* want to do). The name of the design pattern is "proxy". I
strongly suggest that you 1/ try to cure the real problem instead of
hacking around and 2/ read about the proxy design pattern.

>

My 2 cents...

>

I like the idea of mix-ins, but can't figure out how to make a proxy
work that way.

>
You mean, "how to use a proxy for lazy initialization" ? Heck, that's
the exact use case in the GoF.

I mean, "how to make a MixIn class that uses the proxy pattern". I'd
like to be able to do something like this:

class SuperCursor(Fea tureOneMixIn, FeatureTwoMixin , ...,
VanillaCursor): pass

This works with my current implementation. After thinking about it
more, I think I've got such a thing written. I had to use
inspect.getmro and new.classobj to do it, but it works and it honors
the usual mro (as far as I can tell). I've put the code at the bottom
to (try to) maintain readability.

For a long time I had a proxy class that added five or
six features on top of the MySQLdb package, but it wasn't configurable
enough, and I'm working on splitting each feature into its own MixIn
class.

>

As an aside, this is working for me pretty well. The "reconnect"
method (inheritied from a "Reconnecta ble" mixin) uses several of the
object's attributes, so I need to set _inited beforehand so that I
don't get into an infinite __getattribute_ _ loop. What I'd *really*
like to do is remove __getattribute_ _ from the object at that point.

>
You can't. Or, more exactly, all you can do is remove __getattribute_ _
from the mixin class - but then the mixin class won't work anymore. I
don't mean to be condescendant, but it looks like you don't have a clear
understanding of Python's object model here - else you wouldn't even
consider doing such a thing. FWIW, I posted a solution based on the
__getattr__ hook, which did work - at least for the "specs" implied by
your code snippet.

My toy example turned out to be not the best representation of the
problem.
The base class has attributes that "exist" but either throw errors or
segfault
if used before reconnect() is called. This means that I need to
capture more than
just the attributes that would throw AttributeError.

#CODE########## ############### ###############
class Base(object):
def __init__(self, *args, **kwargs):
self.args = args
self.kwargs = kwargs
def __str__(self): return "<Base object created with %s %s>" %
(self.args, self.kwargs)

class MixIn2(object):
def __str__(self):
return "<MixIn2 with %s>" % super(MixIn2, self).__str__()

class MixIn1(object):
def __str__(self):
return "<MixIn1 with %s>" % super(MixIn1, self).__str__()

class ProxyMixIn(obje ct):
def __init__(self, *args, **kwargs):
self.__proxied = None
self.__args = args
self.__kwargs = kwargs
def __getattr__(sel f, attr):
print "Getting", attr
try: return getattr(self.__ proxied, attr)
except AttributeError:
if self.__proxied: raise
else:
from inspect import getmro
mro = getmro(self.__c lass__)
mro = mro[list(mro).index (ProxyMixIn) + 1:]
print "Proxied mro", mro
from new import classobj
self.__proxied = classobj("Proxi ed", mro, globals())
(*self.__args, **self.__kwargs )
return getattr(self.__ proxied, attr)
def __str__(self):
return "<Proxy of %s>" % super(ProxyMixI n, self).__str__()

class Proxy(MixIn1, ProxyMixIn, MixIn2, Base): pass

def main():
p = Proxy(1,2,3, one=1, two=2)
print p
main()

#OUTPUT######## ############### ###############
Getting args
Proxied mro (<class '__main__.MixIn 2'>, <class '__main__.Base' >, <type
'object'>)
Getting kwargs
<MixIn1 with <Proxy of <MixIn2 with <Base object created with (1, 2,
3) {'two': 2, 'one': 1}>>>>

**Carl Banks** · Sep 4 '08, 07:25 PM

Re: Late initialization using __getattribute_ _

On Sep 4, 12:36 pm, bukzor <workithar...@g mail.comwrote:

>>so unfortunately I think I need to use __getattribute_ _
>>to do this. I'm doing all this just to make the connection not
>>actually connect until used.
>I may be dumb, but I don't get how this is supposed to solve your
>problem. But anyway : there's a known design pattern for what you're
>trying to do, that doesn't require mixins nor messing with
>__getattribute __ (which, I repeat, is more often than not something you
>*don't* want to do). The name of the design pattern is "proxy". I
>strongly suggest that you 1/ try to cure the real problem instead of
>hacking around and 2/ read about the proxy design pattern.

>

>My 2 cents...

>

I like the idea of mix-ins, but can't figure out how to make a proxy
work that way.

>

You mean, "how to use a proxy for lazy initialization" ? Heck, that's
the exact use case in the GoF.

>
I mean, "how to make a MixIn class that uses the proxy pattern".

You don't. That's not how proxies work.

I'd like to be able to do something like this:
>
class SuperCursor(Fea tureOneMixIn, FeatureTwoMixin , ...,
VanillaCursor): pass

Why does it have to look like that? A good programmer lets the code
look however it has to look to most effectively do it's job.

With a proxy, the "base class" isn't a base class but a member. Here
is a very simple example:

class SuperCursor(obj ect):
def __init__(self):
self._cursor = VanillaCursor()
self._connected = False
def __getattr__(sel f,attr):
if not self._connected :
self._cursor.co nnect()
self._connected = True
return getattr(self._c ursor,attr)

cursor = SuperCursor()

That doens't use a mixin, but why should it?

Carl Banks

**bukzor** · Sep 4 '08, 07:45 PM

Re: Late initialization using __getattribute_ _

I'd like to be able to do something like this:

class SuperCursor(Fea tureOneMixIn, FeatureTwoMixin , ...,
VanillaCursor): pass

>
Why does it have to look like that? A good programmer lets the code
look however it has to look to most effectively do it's job.
>
With a proxy, the "base class" isn't a base class but a member. Here
is a very simple example:
>
class SuperCursor(obj ect):
def __init__(self):
self._cursor = VanillaCursor()
self._connected = False
def __getattr__(sel f,attr):
if not self._connected :
self._cursor.co nnect()
self._connected = True
return getattr(self._c ursor,attr)
>
cursor = SuperCursor()
>
That doesn't use a mixin, but why should it?

The point of using a mixin is to not limit myself to inheriting from
VanillaCursor. I want to put this on top of various subclasses of the
vanilla cursor, like TimeLimitedCurs or or RetryingCursor. I have four
other mixins that operate this way, so it's desirable to keep this one
in line with that.

**Carl Banks** · Sep 4 '08, 08:25 PM

Re: Late initialization using __getattribute_ _

On Sep 4, 3:38 pm, bukzor <workithar...@g mail.comwrote:

The point of using a mixin is to not limit myself to inheriting from
VanillaCursor. I want to put this on top of various subclasses of the
vanilla cursor, like TimeLimitedCurs or or RetryingCursor. I have four
other mixins that operate this way, so it's desirable to keep this one
in line with that.

ERROR: The request could not be satisfied

http://www.bartleby.com/59/3/foolishconsi.html

I think that desire is hurting you more than it's helping. It's fine
to be consistent for consistency's sake, but you are taking
consistency to an unhealthy extreme. A mixin is simply the wrong tool
to do this with.

My advice: either use a proxy, or manage your connections better.

Carl Banks

Late initialization using getattribute