Generator inside a class prevent __del__ ??

[Emmanuel]

Hi,

I run across this problem, and couldn't find any solution (python 2.2.2)
:

Code :
===========
from __future__ import generators
[color=blue][color=green][color=darkred]
>>> class titi:[/color][/color][/color]
def __init__(self):
print "init"
def __del__(self):
print "del"
def Gen(self):
yield 1
[color=blue][color=green][color=darkred]
>>> c = titi()[/color][/color][/color]
init[color=blue][color=green][color=darkred]
>>> c = [][/color][/color][/color]
del
==============
Here, everything is normal...
But creating a generator :

Code :
===========
[color=blue][color=green][color=darkred]
>>> class toto:[/color][/color][/color]
def __init__(self):
print "init"
self.Coroutine = self.Gen()
def __del__(self):
print "del"
def Gen(self):
yield 1
[color=blue][color=green][color=darkred]
>>> a = toto()[/color][/color][/color]
init[color=blue][color=green][color=darkred]
>>> c = [][/color][/color][/color]
<--- Nothing there !!!
==============

I can't understand why the destructor is not called when a generator is
created, and what I should do to have a "correct" behavior.
(perhaps I missed something obvious, but I can't find it )
Thank you for any help,

Emmanuel

[Terry Reedy]

Re: Generator inside a class prevent __del__ ??


"Emmanuel" <eastier@free.f r> wrote in message
news:4085BA96.6 51D2E4A@free.fr ...[color=blue]
> I run across this problem, and couldn't find any solution (python 2.2.2)[/color]
....[color=blue]
> Here, everything is normal...
> But creating a generator :[/color]

You both defined generator function and called it to create generator
iterator.
[color=blue]
>
> Code :
> ===========
>[color=green][color=darkred]
> >>> class toto:[/color][/color]
> def __init__(self):
> print "init"
> self.Coroutine = self.Gen()[/color]

This creates a reference loop. Delete this (and correct typo below) and
'problem' will disappear.
[color=blue]
> def __del__(self):
> print "del"
> def Gen(self):[/color]

If you do not really use self in the resulting iterator, define this
outside of the class without self as a parameter, and problem will
disappear.
[color=blue]
> yield 1
>[color=green][color=darkred]
> >>> a = toto()[/color][/color][/color]

did you mean 'c = toto()'?
[color=blue]
> init[color=green][color=darkred]
> >>> c = [][/color][/color]
> <--- Nothing there !!!
> ==============
>
> I can't understand why the destructor is not called when a generator is
> created, and what I should do to have a "correct" behavior.[/color]

Either do not create reference loop or break it with del c.Coroutine.

Terry J. Reedy

[Mark Day]

Re: Generator inside a class prevent __del__ ??

In article <4085BA96.651D2 E4A@free.fr>, Emmanuel <eastier@free.f r>
wrote:
[color=blue][color=green][color=darkred]
> >>> class toto:[/color][/color]
> def __init__(self):
> print "init"
> self.Coroutine = self.Gen()
> def __del__(self):
> print "del"
> def Gen(self):
> yield 1
>[color=green][color=darkred]
> >>> a = toto()[/color][/color]
> init[color=green][color=darkred]
> >>> c = [][/color][/color]
> <--- Nothing there !!![/color]

First of all, "a" is still referencing your toto object. I think you
meant "a = []" here. But even if you did "a = []", the destructor
still isn't called. There must still be a reference to the object. My
guess is that the generator (directly or indirectly) is referencing the
object, creating a self referential loop.

Consider the following modification that merely references a function,
and does not create a generator:
[color=blue][color=green][color=darkred]
>>> class tata:[/color][/color][/color]
.... def __init__(self):
.... print "init"
.... self.Coroutine = self.Gen
.... def __del__(self):
.... print "del"
.... def Gen(self):
.... pass
....[color=blue][color=green][color=darkred]
>>> a=tata()[/color][/color][/color]
init[color=blue][color=green][color=darkred]
>>> a=[]
>>>[/color][/color][/color]

Here's how to break that loop:
[color=blue][color=green][color=darkred]
>>> b=tata()[/color][/color][/color]
init[color=blue][color=green][color=darkred]
>>> b.Coroutine=Non e
>>> b=[][/color][/color][/color]
del[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

-Mark

[Rob Nikander]

Re: Generator inside a class prevent __del__ ??

Mark Day wrote:[color=blue]
> still isn't called. There must still be a reference to the object. My
> guess is that the generator (directly or indirectly) is referencing the
> object, creating a self referential loop.
>[/color]

Python has a garbage collector that will try to find these objects with
cyclic references.

from test import *[color=blue][color=green][color=darkred]
>>> a = toto()[/color][/color][/color]
init[color=blue][color=green][color=darkred]
>>> a = None
>>> import gc
>>> gc.garbage[/color][/color][/color]
[][color=blue][color=green][color=darkred]
>>> gc.collect()[/color][/color][/color]
4[color=blue][color=green][color=darkred]
>>> gc.garbage[/color][/color][/color]
[<test.toto instance at 0x81cb78c>][color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

I checked out the documentation for that gc.garbage list and it says
that the collector can't free objects in cycles if the cyles have
objects that have __del__ methods. So it puts them in this list.

I wonder what other garbage collectors do in this situation? Anyone
know? Java?

Rob

[Duncan Booth]

Re: Generator inside a class prevent __del__ ??

Rob Nikander <rnikaREMOVEnde r@adelphia.net> wrote in
news:kNKdnYsIEu HbdRjdRVn-uA@adelphia.com :
[color=blue]
> I checked out the documentation for that gc.garbage list and it says
> that the collector can't free objects in cycles if the cyles have
> objects that have __del__ methods. So it puts them in this list.
>
> I wonder what other garbage collectors do in this situation? Anyone
> know? Java?
>[/color]

Most garbage collectors will do peculiar things if you have destructors or
finalizers in the objects. The problem is that if two objects with
finalizers reference each other there is no correct order to release the
objects that will guarantee that the other object still exists, so the
system either has to choose an arbitrary order, or refuse to call the
finalizers.

The .Net garbage collector is typical. Objects may have finalizers, and
these finalizers are called as part of the garbage collection. The system
guarantees that any finalizer is called exactly 0 or more times --- usually
it is called once when the object is garbage collected, but if the object
is never collected it may not be called at all, and if the object
resurrects itself (e.g. during the finalizer it assigns itself to a global
variable) the finalizer could be called more than once.

A separate thread pool is used for finalizers, so your finalizer could be
called while a user thread is executing a method on the object, and two
objects which refer to each other could have their finalizers called in any
order, or even simultaneously on separate threads. Effectively, this makes
finalizers useless in all but the most obscure situations.

When resources need to be released you should try to do it explicitly. In
..Net this is handled by the Dispose() method, and the finalizer can then
either try calling Dispose() if it has not yet been called, or could try
logging an error although even that may be problematic from a finalizer.

[Emmanuel]

Re: Generator inside a class prevent __del__ ??



Terry Reedy a écrit :
[color=blue]
> "Emmanuel" <eastier@free.f r> wrote in message
> news:4085BA96.6 51D2E4A@free.fr ...[color=green]
> > I run across this problem, and couldn't find any solution (python 2.2.2)[/color]
> ...[color=green]
> > Here, everything is normal...
> > But creating a generator :[/color]
>
> You both defined generator function and called it to create generator
> iterator.[/color]

Yes, I don't have all the generators vocabulary yet...
[color=blue]
>
>[color=green]
> >
> > Code :
> > ===========
> >[color=darkred]
> > >>> class toto:[/color]
> > def __init__(self):
> > print "init"
> > self.Coroutine = self.Gen()[/color]
>
> This creates a reference loop. Delete this (and correct typo below) and
> 'problem' will disappear.[/color]
[color=blue]
>[color=green]
> > def __del__(self):
> > print "del"
> > def Gen(self):[/color]
>
> If you do not really use self in the resulting iterator, define this
> outside of the class without self as a parameter, and problem will
> disappear.[/color]

I didn't use self in order to provide a simple example. In my real class, self
is used...
[color=blue]
>
>[color=green]
> > yield 1
> >[color=darkred]
> > >>> a = toto()[/color][/color]
>
> did you mean 'c = toto()'?[/color]

Yes, sorry for that...
[color=blue]
>
>[color=green]
> > init[color=darkred]
> > >>> c = [][/color]
> > <--- Nothing there !!!
> > ==============
> >
> > I can't understand why the destructor is not called when a generator is
> > created, and what I should do to have a "correct" behavior.[/color]
>
> Either do not create reference loop or break it with del c.Coroutine.
>
> Terry J. Reedy[/color]

Thank you very much for your answer, but I'm still not sure I understand it.
If I understand your words right, creating self.Coroutine as an iterator on
the generator function will create a reference on self, so if I want to use a
generator in a class ( and I really want to ), I must delete explicitly the
iterator before I destroy the object.

Trouble is, I _would_ like not to care about the lifetime of the object, and I
don't know where it will be destroyed.
Should I encapsulate this object in another one, like this :

import toto

class TotoCapsule:
def __init__( self ):
self.toto = toto.toto()
def __del__(self):
del self.toto.Corou tine
self.toto = None

And use TotoCapsule ?
But it means I have to write a lot of more code to access toto's method.
Is there a pattern I missed to dea l with that ?

Thanks a lot,

Emmanuel

[Terry Reedy]

Re: Generator inside a class prevent __del__ ??


"Emmanuel" <eastier@free.f r> wrote in message
news:40866ECD.A 45758B9@free.fr ...[color=blue]
>
>
> Terry Reedy a écrit :[color=green][color=darkred]
> > > >>> class toto:
> > > def __init__(self):
> > > print "init"
> > > self.Coroutine = self.Gen()[/color]
> >
> > This creates a reference loop. Delete this (and correct typo below)[/color][/color]
and[color=blue][color=green]
> > 'problem' will disappear.[/color][/color]

To amplify: the usual idiom for an instance-associated generator is to name
the generator function (method) __iter__ (with one param, self) and to
create and get a reference to the generator via iter() or let the for loop
mechanism do so for you.

c = C(*args)
cgen =iter(c)

Then there is no reference loop. And you can pass around the cgen object
just like any other. If you only need the instance after initialization to
get the generator and you only need one generator for the instance, then
combine the two lines into

cgen = iter(C(*args))

and the *only* reference to the instance is the one in the generator, which
will disappear at the end of a for loop or with an explicit 'del cgen'.

There is also the question whether you actually *need* to get rid of the
object while the program is still running instead of just letting the
program finish and clean up.

Terry J. Reedy

[Andrew Bennetts]

Re: Generator inside a class prevent __del__ ??

On Wed, Apr 21, 2004 at 02:53:33PM +0200, Emmanuel wrote:[color=blue]
>
> Trouble is, I _would_ like not to care about the lifetime of the object, and I
> don't know where it will be destroyed.[/color]

Then don't use __del__. Python can and will automatically collect cycles
when the objects *don't* define __del__ methods.

Out of curiousity, why are you defining __del__ anyway?

-Andrew.

[Emmanuel]

Re: Generator inside a class prevent __del__ ??



Andrew Bennetts a écrit :
[color=blue]
> On Wed, Apr 21, 2004 at 02:53:33PM +0200, Emmanuel wrote:[color=green]
> >
> > Trouble is, I _would_ like not to care about the lifetime of the object, and I
> > don't know where it will be destroyed.[/color]
>
> Then don't use __del__. Python can and will automatically collect cycles
> when the objects *don't* define __del__ methods.
>
> Out of curiousity, why are you defining __del__ anyway?
>
> -Andrew.[/color]



I don't want to use __del__, but I suspected I had an issue with the destruction of
my objects, and used a log in __del__ to monitor the destruction.

But defining __del__ has also a lot of valuable utilisation, or so I think...

Emmanuel

[Joe Mason]

Re: Generator inside a class prevent __del__ ??

In article <40866ECD.A4575 8B9@free.fr>, Emmanuel wrote:[color=blue]
> Thank you very much for your answer, but I'm still not sure I understand it.
> If I understand your words right, creating self.Coroutine as an iterator on
> the generator function will create a reference on self, so if I want to use a
> generator in a class ( and I really want to ), I must delete explicitly the
> iterator before I destroy the object.
>
> Trouble is, I _would_ like not to care about the lifetime of the object, and I
> don't know where it will be destroyed.[/color]

Try looking up "weakref". (I've never used them myself, so I don't know
the exact syntax.)

Joe

[Andrew Bennetts]

Re: Generator inside a class prevent __del__ ??

On Thu, Apr 22, 2004 at 01:26:18AM +0200, Emmanuel wrote:[color=blue]
>
>
> Andrew Bennetts a écrit :
>[color=green]
> > On Wed, Apr 21, 2004 at 02:53:33PM +0200, Emmanuel wrote:[color=darkred]
> > >
> > > Trouble is, I _would_ like not to care about the lifetime of the object, and I
> > > don't know where it will be destroyed.[/color]
> >
> > Then don't use __del__. Python can and will automatically collect cycles
> > when the objects *don't* define __del__ methods.
> >
> > Out of curiousity, why are you defining __del__ anyway?
> >
> > -Andrew.[/color]
>
> I don't want to use __del__, but I suspected I had an issue with the destruction of
> my objects, and used a log in __del__ to monitor the destruction.[/color]

Except that __del__ affects how they are destructed :)

Weakrefs are probably a better choice for this, as they don't interfere with
the lifecycle of the object you're interested in, unlike __del__.
[color=blue]
> But defining __del__ has also a lot of valuable utilisation, or so I think...[/color]

It's only very very rarely useful, in my experience. Again, weakrefs are
probably more useful for what you have in mind.

-Andrew.

[Emmanuel]

Re: Generator inside a class prevent __del__ ??



Andrew Bennetts a écrit :
[color=blue]
> On Thu, Apr 22, 2004 at 01:26:18AM +0200, Emmanuel wrote:[color=green]
> >
> >
> > Andrew Bennetts a écrit :
> >[color=darkred]
> > > On Wed, Apr 21, 2004 at 02:53:33PM +0200, Emmanuel wrote:
> > > >
> > > > Trouble is, I _would_ like not to care about the lifetime of the object, and I
> > > > don't know where it will be destroyed.
> > >
> > > Then don't use __del__. Python can and will automatically collect cycles
> > > when the objects *don't* define __del__ methods.
> > >
> > > Out of curiousity, why are you defining __del__ anyway?
> > >
> > > -Andrew.[/color]
> >
> > I don't want to use __del__, but I suspected I had an issue with the destruction of
> > my objects, and used a log in __del__ to monitor the destruction.[/color]
>
> Except that __del__ affects how they are destructed :)
>
> Weakrefs are probably a better choice for this, as they don't interfere with
> the lifecycle of the object you're interested in, unlike __del__.
>[color=green]
> > But defining __del__ has also a lot of valuable utilisation, or so I think...[/color]
>
> It's only very very rarely useful, in my experience. Again, weakrefs are
> probably more useful for what you have in mind.
>
> -Andrew.[/color]

Ok, I think I don't understand anything anymore...

I thought __del__ was the destructor of the object, like the object::~object in C++ ( my
experience in programming is mainly from C++ ), and so __del__ shouldn't affect when they
are destructed.
And I thought weakref is a way to control the lifetime, ie when the ref count is
decremented, and when to call __del__.

From what you ( and others ) are saying, I'm proven wrong...

Do you know where I can find more information, beside python doc ?

Thanks,

Emmanuel

[Emmanuel]

Re: Generator inside a class prevent __del__ ??



Terry Reedy a écrit :
[color=blue]
> "Emmanuel" <eastier@free.f r> wrote in message
> news:40866ECD.A 45758B9@free.fr ...[color=green]
> >
> >
> > Terry Reedy a écrit :[color=darkred]
> > > > >>> class toto:
> > > > def __init__(self):
> > > > print "init"
> > > > self.Coroutine = self.Gen()
> > >
> > > This creates a reference loop. Delete this (and correct typo below)[/color][/color]
> and[color=green][color=darkred]
> > > 'problem' will disappear.[/color][/color]
>
> To amplify: the usual idiom for an instance-associated generator is to name
> the generator function (method) __iter__ (with one param, self) and to
> create and get a reference to the generator via iter() or let the for loop
> mechanism do so for you.
>
> c = C(*args)
> cgen =iter(c)
>
> Then there is no reference loop. And you can pass around the cgen object
> just like any other. If you only need the instance after initialization to
> get the generator and you only need one generator for the instance, then
> combine the two lines into
>
> cgen = iter(C(*args))
>
> and the *only* reference to the instance is the one in the generator, which
> will disappear at the end of a for loop or with an explicit 'del cgen'.[/color]

But I obviously need other references to my object in my code, my object isn't
modified by the generator only.
I want to resume my generator from time to time during the execution of my app,
and to modify the members of the objects somewhere else ( interaction between
my objects ).
Doing this result in my nicer programmation style than without generators.

[color=blue]
>
>
> There is also the question whether you actually *need* to get rid of the
> object while the program is still running instead of just letting the
> program finish and clean up.
>[/color]

I have a _lot_ of objects created whenever they want, and I don't know where
they will finish their job.
Additionnaly, I'm not developping only on PC, but also on platforms where there
is not so much memory avalaible.

By the way, it seems I still have a lot to understand on this subject.
Do you know any link, example, or whatever, that I could have a look at ?

Thank you very much for your answers,

Emmanuel

[Andrew Bennetts]

Re: Generator inside a class prevent __del__ ??

On Thu, Apr 22, 2004 at 11:12:27AM +0200, Emmanuel wrote:[color=blue]
>
> Ok, I think I don't understand anything anymore...
>
> I thought __del__ was the destructor of the object, like the object::~object in C++ ( my
> experience in programming is mainly from C++ ), and so __del__ shouldn't affect when they
> are destructed.[/color]

__del__ unfortunately *does* impact the lifetime of the object, at least in
CPython:

http://docs.python.org/lib/module-gc.html#l2h-403

http://docs.python.org/ref/customization.html#l2h-175

It's main use used to be to break reference cycles, because before Python
2.0 (or perhaps 1.6?), it couldn't automatically collect reference cycles
because it used a purely ref-count based approach. Now that cycles are
automatically collected, there's not much point in defining __del__ (and it
can actually have unexpected results).
[color=blue]
> And I thought weakref is a way to control the lifetime, ie when the ref count is
> decremented, and when to call __del__.[/color]

No -- weakref doesn't affect the lifetime, that's it's point. It's a way to
have a reference to an object that doesn't keep the object alive if nothing
else is. As the documentation at
http://docs.python.org/lib/module-weakref.html says:

A weak reference to an object is not enough to keep the object alive:
when the only remaining references to a referent are weak references,
garbage collection is free to destroy the referent and reuse its memory
for something else.
[color=blue][color=green]
> >From what you ( and others ) are saying, I'm proven wrong...[/color]
>
> Do you know where I can find more information, beside python doc ?[/color]

Try googling for tutorials and things, there's probably stuff out there.
The Python docs are pretty good, though... the weakref module has pretty
comprehensive documentation, and the description of __del__ in the language
reference has big note that mentions that garbage-collection of cycles
doesn't work when __del__ methods are involved.

I've also found books such as Python in a Nutshell and the Python Essential
Reference to be quite good at pointing this sort of thing out, when I've
looked. I usually rely on the official Python docs, though.

-Andrew.