Garbage collector and threads

[Nicolas Fleury]

Hi everyone,
I just discovered, correct me if I'm wrong, that as long as a
threading.Threa d is running, its destructor will not be called, because
it is referring to himself. So if I have something like:

class MyThread(thread ing.Thread):
def __init__(self):
self.cancelEven t = threading.Event ()
threading.Threa d.__init__(self )
def __del__(self):
self.cancel()
def run(self):
self.cancelEven t.wait()
def cancel(self):
self.cancelEven t.set()

I must call cancel from the outside if I want the destructor to be
called (note that I don't want deamon threads). I can make a wrapper
over threading.Threa d to have the behaviour I want: have a thread with
__del__ called when it is not referred by another thread. But my
question is, and I don't have an overall vision of the issue at all,
should that be the default behaviour anyway?

Regards,
Nicolas

[Aahz]

Re: Garbage collector and threads

In article <hgJ0c.4318$mx4 .105259@nnrp1.u unet.ca>,
Nicolas Fleury <nid_oizo@yahoo .com_remove_the _> wrote:[color=blue]
>
> I just discovered, correct me if I'm wrong, that as long as a
>threading.Thre ad is running, its destructor will not be called, because
>it is referring to himself. So if I have something like:
>
>class MyThread(thread ing.Thread):
> def __init__(self):
> self.cancelEven t = threading.Event ()
> threading.Threa d.__init__(self )
> def __del__(self):
> self.cancel()
> def run(self):
> self.cancelEven t.wait()
> def cancel(self):
> self.cancelEven t.set()
>
>I must call cancel from the outside if I want the destructor to be
>called (note that I don't want deamon threads). I can make a wrapper
>over threading.Threa d to have the behaviour I want: have a thread with
>__del__ called when it is not referred by another thread. But my
>question is, and I don't have an overall vision of the issue at all,
>should that be the default behaviour anyway?[/color]

Yes, Don't Do That. Do not rely on finalizers for external resources
(such as threads).

Note that your understanding of Python's memory management could use some
improvement: GC *never* runs on objects containing __del__
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

"Do not taunt happy fun for loops. Do not change lists you are looping over."
--Remco Gerlich, comp.lang.pytho n

[Nicolas Fleury]

Re: Garbage collector and threads

Aahz wrote:[color=blue]
> Yes, Don't Do That. Do not rely on finalizers for external resources
> (such as threads).[/color]

Why? Isn't a kind of RAII applied to Python?
[color=blue]
> Note that your understanding of Python's memory management could use some
> improvement: GC *never* runs on objects containing __del__[/color]

Well, I guess you're right. What I mean is the reasons behind the call
to __del__, whatever it is ref-counting or something else. Wouldn't a
class like CancellableThre ad, with a sub-class implemented cancel method
called when the object is not referred in any other thread be useful?

Regards,
Nicolas

[Aahz]

Re: Garbage collector and threads

In article <1EJ0c.4334$mx4 .105438@nnrp1.u unet.ca>,
Nicolas Fleury <nid_oizo@yahoo .com_remove_the _> wrote:[color=blue]
>Aahz wrote:[color=green]
>>
>> Yes, Don't Do That. Do not rely on finalizers for external resources
>> (such as threads).[/color]
>
>Why? Isn't a kind of RAII applied to Python?[/color]

What's "RAII"?
[color=blue][color=green]
>> Note that your understanding of Python's memory management could use some
>> improvement: GC *never* runs on objects containing __del__[/color]
>
>Well, I guess you're right. What I mean is the reasons behind the call
>to __del__, whatever it is ref-counting or something else. Wouldn't a
>class like CancellableThre ad, with a sub-class implemented cancel method
>called when the object is not referred in any other thread be useful?[/color]

Perhaps, but threading needs references to created threads in order to
implement its functionality. You might try building your own thread
constructs using ``thread`` directly.
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

"Do not taunt happy fun for loops. Do not change lists you are looping over."
--Remco Gerlich, comp.lang.pytho n

[Michael Hudson]

Re: Garbage collector and threads

aahz@pythoncraf t.com (Aahz) writes:
[color=blue]
> In article <1EJ0c.4334$mx4 .105438@nnrp1.u unet.ca>,
> Nicolas Fleury <nid_oizo@yahoo .com_remove_the _> wrote:[color=green]
> >Aahz wrote:[color=darkred]
> >>
> >> Yes, Don't Do That. Do not rely on finalizers for external resources
> >> (such as threads).[/color]
> >
> >Why? Isn't a kind of RAII applied to Python?[/color]
>
> What's "RAII"?[/color]

Resource Aquisition Is Initialization. Works in C++. Doesn't really
in Python.

Cheers,
mwh

--
Need to Know is usually an interesting UK digest of things that
happened last week or might happen next week. [...] This week,
nothing happened, and we don't care.
-- NTK Now, 2000-12-29, http://www.ntk.net/

[Nicolas Fleury]

Re: Garbage collector and threads

Aahz wrote:[color=blue]
> What's "RAII"?[/color]

"Resource Acquisition Is Initialization"
It's a term frequently used in C++ to describe the use of constructor
and destructor for resource allocation/deallocation.
[color=blue][color=green]
>>to __del__, whatever it is ref-counting or something else. Wouldn't a
>>class like CancellableThre ad, with a sub-class implemented cancel method
>>called when the object is not referred in any other thread be useful?[/color]
>
>
> Perhaps, but threading needs references to created threads in order to
> implement its functionality. You might try building your own thread
> constructs using ``thread`` directly.[/color]

I still don't know how to implement it. I can do something very simple
like:

class Canceller:
def __init__(self, thread):
self.thread = thread
def __del__(self):
self.thread.can cel()
self.thread.joi n()

but I can't merge the concept with a Thread class, because the self
passed to run would be a reference to the object... I don't know if a
language extension would be worth the effort or if there's a cool
solution I'm missing, but what is sure is that I write a lot of code to
cancel threads in finalizers of parent objects. I don't want deamon
threads so that everything is stopped cleanly, but at the same time it
is error-prone in a GUI application, because I risk to have the process
still running after a quit.

I just want a way to garantee that all my threads are cancelled when the
main thread is finished.

Regards,
Nicolas

[Aahz]

Re: Garbage collector and threads

In article <6uK0c.4347$mx4 .106055@nnrp1.u unet.ca>,
Nicolas Fleury <nid_oizo@yahoo .com_remove_the _> wrote:[color=blue]
>Aahz wrote:[color=green]
>>
>> What's "RAII"?[/color]
>
>"Resource Acquisition Is Initialization"
>It's a term frequently used in C++ to describe the use of constructor
>and destructor for resource allocation/deallocation.[/color]

As Michael Hudson said, it doesn't work that way in Python, because all
Python objects are global (and I mean truly global here, not just module
global). Essentially, all objects are heap objects that can be traced
by crawling through Python's internals.
[color=blue][color=green]
>> Perhaps, but threading needs references to created threads in order to
>> implement its functionality. You might try building your own thread
>> constructs using ``thread`` directly.[/color]
>
>I still don't know how to implement it. I can do something very simple
>like:
>
>class Canceller:
> def __init__(self, thread):
> self.thread = thread
> def __del__(self):
> self.thread.can cel()
> self.thread.joi n()
>
>but I can't merge the concept with a Thread class, because the self
>passed to run would be a reference to the object... I don't know if a
>language extension would be worth the effort or if there's a cool
>solution I'm missing, but what is sure is that I write a lot of code to
>cancel threads in finalizers of parent objects. I don't want deamon
>threads so that everything is stopped cleanly, but at the same time it
>is error-prone in a GUI application, because I risk to have the process
>still running after a quit.[/color]

Right. As I said, you have to use the ``thread`` module directly; you
can't use ``threading``.
[color=blue]
>I just want a way to garantee that all my threads are cancelled when the
>main thread is finished.[/color]

So hold references to them and call the ``cancel()`` method yourself.
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

"Do not taunt happy fun for loops. Do not change lists you are looping over."
--Remco Gerlich, comp.lang.pytho n

[Nicolas Fleury]

Re: Garbage collector and threads

Aahz wrote:[color=blue]
> As Michael Hudson said, it doesn't work that way in Python, because all
> Python objects are global (and I mean truly global here, not just module
> global). Essentially, all objects are heap objects that can be traced
> by crawling through Python's internals.[/color]

I understand that. RAII is often used when speaking of scoped
variables. In my case I was talking about global scope in main thread.
[color=blue]
> Right. As I said, you have to use the ``thread`` module directly; you
> can't use ``threading``.[/color]

But even if I use "thread", I have no way to put the functionality in
one class if I want access the members in the running thread.
[color=blue][color=green]
>>I just want a way to garantee that all my threads are cancelled when the
>>main thread is finished.[/color]
>
> So hold references to them and call the ``cancel()`` method yourself.[/color]

That's a possible solution. Make a custom Thread class with a cancel
method and with all instances registered somewhere. At the end of the
main thread, I call something to cancel all these threads. There's
however some problems with that solution:
- Parent threads need to be cancelled before their children threads.
- It only works at the end of the main thread; it cannot be used at the
middle of an appplication.
- The cancellations are not automatic (I guess there's a way to do it by
overriding something like exit, anyway...)

All these problems are solved by using RAII with the Canceller class I
shown in a previous message, so I prefer using this solution. In fact,
what I would really need, is a mechanism, and I don't know if there's
already one, to disable ref-counting of a thread object in its
corresponding thread, so that I could provide a single class for
cancellable threads.

Regards,
Nicolas

[Aahz]

Re: Garbage collector and threads

In article <R4M0c.4365$mx4 .107234@nnrp1.u unet.ca>,
Nicolas Fleury <nid_oizo@yahoo .com_remove_the _> wrote:[color=blue]
>Aahz wrote:[color=green]
>>
>> Right. As I said, you have to use the ``thread`` module directly; you
>> can't use ``threading``.[/color]
>
>But even if I use "thread", I have no way to put the functionality in
>one class if I want access the members in the running thread.[/color]

Huh? I don't understand what you mean.
[color=blue][color=green][color=darkred]
>>>I just want a way to garantee that all my threads are cancelled when the
>>>main thread is finished.[/color]
>>
>> So hold references to them and call the ``cancel()`` method yourself.[/color]
>
>That's a possible solution. Make a custom Thread class with a cancel
>method and with all instances registered somewhere. At the end of the
>main thread, I call something to cancel all these threads. There's
>however some problems with that solution:
>- Parent threads need to be cancelled before their children threads.[/color]

Why's that a problem? Just hold references to the parent threads in the
main thread and cascade the cancels down.
[color=blue]
>- It only works at the end of the main thread; it cannot be used at the
>middle of an appplication.[/color]

Why?
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

"Do not taunt happy fun for loops. Do not change lists you are looping over."
--Remco Gerlich, comp.lang.pytho n

[Nicolas Fleury]

Re: Garbage collector and threads

Aahz wrote:[color=blue]
> In article <R4M0c.4365$mx4 .107234@nnrp1.u unet.ca>,
> Nicolas Fleury <nid_oizo@yahoo .com_remove_the _> wrote:[color=green]
>>But even if I use "thread", I have no way to put the functionality in
>>one class if I want access the members in the running thread.[/color]
>
> Huh? I don't understand what you mean.[/color]

Well, suppose I want to have a class with the same interface of
threading.Threa d, but with two additions:
class CancellableThre ad:
def __del__(self): self.cancel()

The first addition is that the finalizer use a sub-class implemented
mechanism for cancellation of the thread.
The second addition is that, as soon as the object is not referred
anymore in the parent thread, the thread object is finalized.

Note that it doesn't need to be the finalizer, what is necessary is
something to be called when the object is no more referred in the parent
thread.

I'm saying that I don't know of any way to do that with a single class,
because I expect the sub-class to define something like:
def run(self)
And the thread object will be referred in the running thread by "self",
so I'm stuck without any way to know when the thread object is not
referred in the main thread.

I can do something like:
class Canceller:
def __init__(self, thread): self.thread = thread
def __del__(self): self.thread.can cel()

so that I encapsulate all threads with "Cancellers " in the parent
thread, but then I have to use two objects to the job that could
possibly be done by one.
[color=blue][color=green]
>>- Parent threads need to be cancelled before their children threads.[/color]
>
> Why's that a problem? Just hold references to the parent threads in the
> main thread and cascade the cancels down.[/color]

You're right, it works. At the moment I wrote that I expected mix of a
CancellableThre ad class implemented with this mechanism with other
threads classes to cause loss of link between threads. If that happens
all threads that are not CancellableThre ads (like the main thread) must
call the function to cancel remaining CancellableThre ads started
directly by them.
[color=blue][color=green]
>>- It only works at the end of the main thread; it cannot be used at the
>>middle of an appplication.[/color]
>
> Why?[/color]

Because the goal is to cancel threads that are no more referred in the
main thread. Since there's no way to know which they are, there's no
way to do it at the middle of an application. You have to do it at the
end of the main thread, as for other threads, where you can assumed that
cancellable threads should be cancelled and joined.

I don't know if my explications of what I'm searching are more clear. I
have the feeling it's impossible to do it without a language extension
and at the same time that it would be less error-prone than
threading.Threa d as deamons or not. Maybe I could write a PEP just for
the exercise of it...

Regards,
Nicolas