Re: How to kill a thread?

[=?ISO-8859-1?Q?Gerhard_H=E4ring?=]

John Dohn wrote:

Hi there,
>
How can I kill a threading.Threa d subclass from MainThread?
>
My threads are waiting for data in Queue.get() method:
class MyThread(thread ing.Thread):
def run(self):
while True:
data = queue.get() # <- here it waits most of the time
... process data
>
From the main thread I start several working threads:
thr = MyThread()
thr.start()
... feed the queue
and at the end for each thread I'd like to do something like thr.kill()
or thr.stop() or thr.destroy() or ... you got the point. I can't figure
out how.
>
Is there a way to do it?

When I do this, I put a special value in the queue (like None) and in
the worker thread, check for the special value and exit if found.

Threads can also be marked as "daemon threads" (see docs for
threading.Threa d objects). This will make the application terminate if
only "daemon threads" are left.

So best would probably be soemthing like

- call setDaemon() when creating worker threads
- ...
- send all worker/daemon threads None via queue
- wait some time, like time.sleep(1.0) , to let daemon exit themselves
- exit main application

-- Gerhard

[The Pythonista]

Re: How to kill a thread?

It's always been my understanding that you can't forcibly kill a thread
in Python (at least not in a portable way). The best you can do is
politely ask it to die, IIRC.

--
code.py: A blog about life, the universe, and Python

http://pythonista.wordpress.com

** Posted from http://www.teranews.com **

[Rhamphoryncus]

Re: How to kill a thread?

On Jun 6, 12:44 pm, The Pythonista <n...@this.time wrote:

It's always been my understanding that you can't forcibly kill a thread
in Python (at least not in a portable way). The best you can do is
politely ask it to die, IIRC.

Inherently, the best you can do in most languages is ask them politely
to die. Otherwise you'll leave locks and various other datastructures
in an inconvenient state, which is too complex to handle correctly.
The exception is certain functional languages, which aren't capable of
having threads and complex state in the same sense.

However, you could go quite far with a standardized mechanism of
politely asking threads to die. For instance, all blocking I/O
functions could be made to support it. This is how cancellation works
in python-safethread.

[David]

Re: How to kill a thread?

Would it be possible for a 3rd-party threading library to an
'interruptible' version of Queue?

The same methods as the normal Queue, but when you call a new .kill()
method on the queue, all the threads locking on the queue get a
"QueueKille d" exception thrown.

It might be as simple as a Queue wrapper where .kill() adds a speciall
Killed value to the queue, and the .get() wrapper throws a QueueKilled
exception (after re-putting Killed) when it reads Killed.

Locking for queues with a limited size is more complicated. Maybe
..kill() can also set an .killed attribute, and pop anything already in
the queue before putting Killed. Then when the put() wrapper unlocks,
it checks if .killed is set, and throws a QueueKilled exception.

One side-effect is that the queue becomes unusable (get & put now
immediately throw QueueKilled), unless you make a .unkill() method.

David

[Nick Craig-Wood]

Re: How to kill a thread?

David <wizzardx@gmail .comwrote:

Would it be possible for a 3rd-party threading library to an
'interruptible' version of Queue?
>
The same methods as the normal Queue, but when you call a new .kill()
method on the queue, all the threads locking on the queue get a
"QueueKille d" exception thrown.
>
It might be as simple as a Queue wrapper where .kill() adds a speciall
Killed value to the queue, and the .get() wrapper throws a QueueKilled
exception (after re-putting Killed) when it reads Killed.

There seem to me to be 2 basic types of threads - I'll call them
client threads and server threads.

A client thread listens on a Queue for its instructions and replies
via a different Queue, or via Queues passed in in its instructions.
That kind of thread can easily have an instruction which means quit.
When it isn't processing instructions there is a nice clean moment for
it to quit. However if the processing takes a long time then there
will be a long wait for the thread to die. This is the scenario
described above.

The server type thread is processing all the time. Every now and
again it sends its results via a Queue. An example of this type of
thread might be one listening on a socket. This type of thread has no
nice moment to listen for instructions to quit as it might well be
blocked on something else (eg listening to a socket). To make this
type of thread kill nicely you have to go back to breaking up your
work into chunks (eg polling the socket asynchronously) and polling
the command queue to wait for your quit instruction which is wastefull
of CPU resources.

So it seems to me that there isn't a good solution to killing python
threads of either type, other than coding them carefully and checking
for quit instructions regularly.

You can kill them using the internal python API and ctypes, but you
run the risk of leaving things in an inconsistent state which is why
that API isn't exposed.

Here is an attempt at a killable thread

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496960

and

http://sebulba.wikispaces.com/recipe+thread2

Read the caveats!

--
Nick Craig-Wood <nick@craig-wood.com-- http://www.craig-wood.com/nick

[Antoon Pardon]

Re: How to kill a thread?

On 2008-06-07, Rhamphoryncus <rhamph@gmail.c omwrote:

On Jun 6, 12:44 pm, The Pythonista <n...@this.time wrote:

>It's always been my understanding that you can't forcibly kill a thread
>in Python (at least not in a portable way). The best you can do is
>politely ask it to die, IIRC.

>
Inherently, the best you can do in most languages is ask them politely
to die. Otherwise you'll leave locks and various other datastructures
in an inconvenient state, which is too complex to handle correctly.
The exception is certain functional languages, which aren't capable of
having threads and complex state in the same sense.

Well it would of course depend on what is considered asking politely?

If one thread could cause an exception being thrown in an other thread,
would this be considered a polite way to ask? Would it be considered
an acceptable way?

--
Antoon Pardon

[Antoon Pardon]

Re: How to kill a thread?

On 2008-06-07, David <wizzardx@gmail .comwrote:

Would it be possible for a 3rd-party threading library to an
'interruptible' version of Queue?
>
The same methods as the normal Queue, but when you call a new .kill()
method on the queue, all the threads locking on the queue get a
"QueueKille d" exception thrown.

I have done something similar. The idea was that threads had to open
a queue before they could access it. If the last thread who had the
queue open in write mode, closed the queue, a reader trying to get
an element from an empty queue would have an "EOInformat ion" exception
raised.

--
Antoon Pardon

[Rhamphoryncus]

Re: How to kill a thread?

On Jun 9, 5:33 am, Antoon Pardon <apar...@forel. vub.ac.bewrote:

On 2008-06-07, Rhamphoryncus <rha...@gmail.c omwrote:
>

On Jun 6, 12:44 pm, The Pythonista <n...@this.time wrote:

It's always been my understanding that you can't forcibly kill a thread
in Python (at least not in a portable way). The best you can do is
politely ask it to die, IIRC.

>

Inherently, the best you can do in most languages is ask them politely
to die. Otherwise you'll leave locks and various other datastructures
in an inconvenient state, which is too complex to handle correctly.
The exception is certain functional languages, which aren't capable of
having threads and complex state in the same sense.

>
Well it would of course depend on what is considered asking politely?
>
If one thread could cause an exception being thrown in an other thread,
would this be considered a polite way to ask? Would it be considered
an acceptable way?

The exception must not be raised until a point explicitly designed as
safe is hit. Otherwise, any function that manipulates data you'll
still use will potentially be buggered. Consider sys.stdout: codecs,
buffering, lots to go wrong.

[Fuzzyman]

Re: How to kill a thread?

On Jun 9, 9:20 pm, Rhamphoryncus <rha...@gmail.c omwrote:

On Jun 9, 5:33 am, Antoon Pardon <apar...@forel. vub.ac.bewrote:
>
>
>

On 2008-06-07, Rhamphoryncus <rha...@gmail.c omwrote:

>

On Jun 6, 12:44 pm, The Pythonista <n...@this.time wrote:
>It's always been my understanding that you can't forcibly kill a thread
>in Python (at least not in a portable way).  The best you can do is
>politely ask it to die, IIRC.

>

Inherently, the best you can do in most languages is ask them politely
to die.  Otherwise you'll leave locks and various other datastructures
in an inconvenient state, which is too complex to handle correctly.
The exception is certain functional languages, which aren't capable of
having threads and complex state in the same sense.

>

Well it would of course depend on what is considered asking politely?

>

If one thread could cause an exception being thrown in an other thread,
would this be considered a polite way to ask? Would it be considered
an acceptable way?

>
The exception must not be raised until a point explicitly designed as
safe is hit.  Otherwise, any function that manipulates data you'll
still use will potentially be buggered.  Consider sys.stdout: codecs,
buffering, lots to go wrong.

Java and .NET both have ways of killing threads. They both work by
raising a 'ThreadAbort' (or similar) exception in the target thread.
In early implementations they both suffered from a similar problem.
You could protect locks (etc) by having a finally block that would
release all resources as needed - but what happens if the thread abort
exception is raised *inside* the finally block?

Java responded by deprecating thread aborting. .NET responded by
ensuring that a thread abort exception would never be raised inside a
finally block - if that happened the exception would only be raised
once the code has left the finally block.

Aborting threads in .NET can be extremely useful. Politely asking a
thread to die is no good if the task the thread is executing is
extremely coarse grained - it may not be able to respond to the
request for some time. If your code is structured correctly
(performing a long running background calculation for example) then
you may *know* that you can kill it without problems, but Python has
no native API to do this.

Michael Foord

http://www.ironpythoninaction.com/

[Rhamphoryncus]

Re: How to kill a thread?

On Jun 9, 2:52 pm, Fuzzyman <fuzzy...@gmail .comwrote:

On Jun 9, 9:20 pm, Rhamphoryncus <rha...@gmail.c omwrote:
>
>
>

On Jun 9, 5:33 am, Antoon Pardon <apar...@forel. vub.ac.bewrote:

>

On 2008-06-07, Rhamphoryncus <rha...@gmail.c omwrote:

>

On Jun 6, 12:44 pm, The Pythonista <n...@this.time wrote:
It's always been my understanding that you can't forcibly kill a thread
in Python (at least not in a portable way). The best you can do is
politely ask it to die, IIRC.

>

Inherently, the best you can do in most languages is ask them politely
to die. Otherwise you'll leave locks and various other datastructures
in an inconvenient state, which is too complex to handle correctly.
The exception is certain functional languages, which aren't capable of
having threads and complex state in the same sense.

>

Well it would of course depend on what is considered asking politely?

>

If one thread could cause an exception being thrown in an other thread,
would this be considered a polite way to ask? Would it be considered
an acceptable way?

>

The exception must not be raised until a point explicitly designed as
safe is hit. Otherwise, any function that manipulates data you'll
still use will potentially be buggered. Consider sys.stdout: codecs,
buffering, lots to go wrong.

>
Java and .NET both have ways of killing threads. They both work by
raising a 'ThreadAbort' (or similar) exception in the target thread.
In early implementations they both suffered from a similar problem.
You could protect locks (etc) by having a finally block that would
release all resources as needed - but what happens if the thread abort
exception is raised *inside* the finally block?
>
Java responded by deprecating thread aborting. .NET responded by
ensuring that a thread abort exception would never be raised inside a
finally block - if that happened the exception would only be raised
once the code has left the finally block.
>
Aborting threads in .NET can be extremely useful. Politely asking a
thread to die is no good if the task the thread is executing is
extremely coarse grained - it may not be able to respond to the
request for some time. If your code is structured correctly
(performing a long running background calculation for example) then
you may *know* that you can kill it without problems, but Python has
no native API to do this.

So how does .NET deal with the sys.stdout corruption? Does it?

If you've carefully written your code to use only safe primitives and
local state (discarded if interrupted) then yes, it could be
interruptible. At this point you could specially mark that block of
code as safe, leaving the vast majority of other code unsafe by
default. Then again, since you're going to the trouble of carefully
designing and auditing your code you could just make it cancellable
like blocking I/O should be - just by polling a flag at key points
(and you're CPU-bound anyway, so it's not expensive.)

The only place I know of that you *need* arbitrary interruption is
hitting CTRL-C in the interactive interpreter. At this point it's a
debugging tool though, so the risk of weirdness is acceptable.

[Antoon Pardon]

Re: How to kill a thread?

On 2008-06-09, Rhamphoryncus <rhamph@gmail.c omwrote:

On Jun 9, 5:33 am, Antoon Pardon <apar...@forel. vub.ac.bewrote:

>On 2008-06-07, Rhamphoryncus <rha...@gmail.c omwrote:
>>

On Jun 6, 12:44 pm, The Pythonista <n...@this.time wrote:
>It's always been my understanding that you can't forcibly kill a thread
>in Python (at least not in a portable way). The best you can do is
>politely ask it to die, IIRC.

>>

Inherently, the best you can do in most languages is ask them politely
to die. Otherwise you'll leave locks and various other datastructures
in an inconvenient state, which is too complex to handle correctly.
The exception is certain functional languages, which aren't capable of
having threads and complex state in the same sense.

>>
>Well it would of course depend on what is considered asking politely?
>>
>If one thread could cause an exception being thrown in an other thread,
>would this be considered a polite way to ask? Would it be considered
>an acceptable way?

>
The exception must not be raised until a point explicitly designed as
safe is hit. Otherwise, any function that manipulates data you'll
still use will potentially be buggered. Consider sys.stdout: codecs,
buffering, lots to go wrong.

I don't see the point. Exceptions are raised now without the ability
of an explicitly designed safe point. If something unexpected happens
your code can raise an exception and leave your data buggered too if
you didn't anticipate it propely.

--
Antoon Pardon

[Rhamphoryncus]

Re: How to kill a thread?

On Jun 10, 1:55 am, Antoon Pardon <apar...@forel. vub.ac.bewrote:

On 2008-06-09, Rhamphoryncus <rha...@gmail.c omwrote:
>
>
>

On Jun 9, 5:33 am, Antoon Pardon <apar...@forel. vub.ac.bewrote:

On 2008-06-07, Rhamphoryncus <rha...@gmail.c omwrote:

>

On Jun 6, 12:44 pm, The Pythonista <n...@this.time wrote:
It's always been my understanding that you can't forcibly kill a thread
in Python (at least not in a portable way). The best you can do is
politely ask it to die, IIRC.

>

Inherently, the best you can do in most languages is ask them politely
to die. Otherwise you'll leave locks and various other datastructures
in an inconvenient state, which is too complex to handle correctly.
The exception is certain functional languages, which aren't capable of
having threads and complex state in the same sense.

>

Well it would of course depend on what is considered asking politely?

>

If one thread could cause an exception being thrown in an other thread,
would this be considered a polite way to ask? Would it be considered
an acceptable way?

>

The exception must not be raised until a point explicitly designed as
safe is hit. Otherwise, any function that manipulates data you'll
still use will potentially be buggered. Consider sys.stdout: codecs,
buffering, lots to go wrong.

>
I don't see the point. Exceptions are raised now without the ability
of an explicitly designed safe point. If something unexpected happens
your code can raise an exception and leave your data buggered too if
you didn't anticipate it propely.

Although in theory you could get any exception at any point, in
practise you shouldn't unless your program is broken. If it is broken
the exceptions shouldn't be caught and should cause the program to
terminate, so the harm is reduced. The exceptions that should happen
(such as IOError) should be from predicted points, and anticipated.

A notable exception is MemoryError, which can show up from anywhere at
any time. Nobody's come up with a solution to that one, so we usually
just let the program die.

Cancelling a thread implies your program will continue. Otherwise
you'd just exit the whole process (either via _exit() or via daemon
threads.)

[Fuzzyman]

Re: How to kill a thread?

On Jun 10, 2:03 am, Rhamphoryncus <rha...@gmail.c omwrote:

On Jun 9, 2:52 pm, Fuzzyman <fuzzy...@gmail .comwrote:
>
>
>

On Jun 9, 9:20 pm, Rhamphoryncus <rha...@gmail.c omwrote:

>

On Jun 9, 5:33 am, Antoon Pardon <apar...@forel. vub.ac.bewrote:

>

On 2008-06-07, Rhamphoryncus <rha...@gmail.c omwrote:

>

On Jun 6, 12:44 pm, The Pythonista <n...@this.time wrote:
>It's always been my understanding that you can't forcibly kill a thread
>in Python (at least not in a portable way).  The best you can do is
>politely ask it to die, IIRC.

>

Inherently, the best you can do in most languages is ask them politely
to die.  Otherwise you'll leave locks and various other datastructures
in an inconvenient state, which is too complex to handle correctly..
The exception is certain functional languages, which aren't capable of
having threads and complex state in the same sense.

>

Well it would of course depend on what is considered asking politely?

>

If one thread could cause an exception being thrown in an other thread,
would this be considered a polite way to ask? Would it be considered
an acceptable way?

>

The exception must not be raised until a point explicitly designed as
safe is hit.  Otherwise, any function that manipulates data you'll
still use will potentially be buggered.  Consider sys.stdout: codecs,
buffering, lots to go wrong.

>

Java and .NET both have ways of killing threads. They both work by
raising a 'ThreadAbort' (or similar) exception in the target thread.
In early implementations they both suffered from a similar problem.
You could protect locks (etc) by having a finally block that would
release all resources as needed - but what happens if the thread abort
exception is raised *inside* the finally block?

>

Java responded by deprecating thread aborting. .NET responded by
ensuring that a thread abort exception would never be raised inside a
finally block - if that happened the exception would only be raised
once the code has left the finally block.

>

Aborting threads in .NET can be extremely useful. Politely asking a
thread to die is no good if the task the thread is executing is
extremely coarse grained - it may not be able to respond to the
request for some time. If your code is structured correctly
(performing a long running background calculation for example) then
you may *know* that you can kill it without problems, but Python has
no native API to do this.

>
So how does .NET deal with the sys.stdout corruption?  Does it?
>

That has never been an issue for us.

If you've carefully written your code to use only safe primitives and
local state (discarded if interrupted) then yes, it could be
interruptible.  At this point you could specially mark that block of
code as safe, leaving the vast majority of other code unsafe by
default.  Then again, since you're going to the trouble of carefully
designing and auditing your code you could just make it cancellable
like blocking I/O should be - just by polling a flag at key points
(and you're CPU-bound anyway, so it's not expensive.)
>
The only place I know of that you *need* arbitrary interruption is
hitting CTRL-C in the interactive interpreter.  At this point it's a
debugging tool though, so the risk of weirdness is acceptable.

We use background threads for long running calculations that we know
are safe to abort. Resources that need protecting we do with finally
blocks which the thread abort honours.

The calculation is 'coarse grained' (it can call into .NET APIs that
can take a relatively long time to return) - so polling for exit
wouldn't work anyway. We also run user code and can't expect their
code to poll for exit conditions.

This system works fine for us. We had fun syncing results back when
the calculation terminates (race conditions if a new one needs to
start just as the old one ends) - but that is the fun of threads in
the first place and nothing to do with how we terminate calculations
early.

Michael Foord

http://www.ironpythoninaction.com/

[Rhamphoryncus]

Re: How to kill a thread?

On Jun 10, 3:41 pm, Fuzzyman <fuzzy...@gmail .comwrote:

On Jun 10, 2:03 am, Rhamphoryncus <rha...@gmail.c omwrote:

So how does .NET deal with the sys.stdout corruption? Does it?

>
That has never been an issue for us.

Of course. It's far more likely to hit the underlying blocked I/O
than the buffer management, unless it's doing a great deal of I/O. Or
alternatively, they're using some other mechanism to prevent
interruption during buffer management.

If you've carefully written your code to use only safe primitives and
local state (discarded if interrupted) then yes, it could be
interruptible. At this point you could specially mark that block of
code as safe, leaving the vast majority of other code unsafe by
default. Then again, since you're going to the trouble of carefully
designing and auditing your code you could just make it cancellable
like blocking I/O should be - just by polling a flag at key points
(and you're CPU-bound anyway, so it's not expensive.)

>

The only place I know of that you *need* arbitrary interruption is
hitting CTRL-C in the interactive interpreter. At this point it's a
debugging tool though, so the risk of weirdness is acceptable.

>
We use background threads for long running calculations that we know
are safe to abort. Resources that need protecting we do with finally
blocks which the thread abort honours.

How does that protect code like this?

f = open('somefile' , 'w')
# interruption here
try:
...
finally:
...

The calculation is 'coarse grained' (it can call into .NET APIs that
can take a relatively long time to return) - so polling for exit
wouldn't work anyway. We also run user code and can't expect their
code to poll for exit conditions.

Do those .NET get interrupted at arbitrary points? Is this
"untrusted" user code also audited for correctness? (Although I'd bet
you simply document what you do and blame the user if something
breaks.)

I'm not saying it can't be made to work in your specific case - it
likely does work well for you. I'm saying it can't work *in
general*. Stretching it out for general use turns those little cracks
into massive canyons. A language needs a general mechanism that does
work - such as a polite cancellation API.

[Antoon Pardon]

Re: How to kill a thread?

On 2008-06-10, Rhamphoryncus <rhamph@gmail.c omwrote:

On Jun 10, 1:55 am, Antoon Pardon <apar...@forel. vub.ac.bewrote:

>On 2008-06-09, Rhamphoryncus <rha...@gmail.c omwrote:
>>
>>
>>

On Jun 9, 5:33 am, Antoon Pardon <apar...@forel. vub.ac.bewrote:
>On 2008-06-07, Rhamphoryncus <rha...@gmail.c omwrote:

>>

On Jun 6, 12:44 pm, The Pythonista <n...@this.time wrote:
>It's always been my understanding that you can't forcibly kill a thread
>in Python (at least not in a portable way). The best you can do is
>politely ask it to die, IIRC.

>>

Inherently, the best you can do in most languages is ask them politely
to die. Otherwise you'll leave locks and various other datastructures
in an inconvenient state, which is too complex to handle correctly.
The exception is certain functional languages, which aren't capable of
having threads and complex state in the same sense.

>>

>Well it would of course depend on what is considered asking politely?

>>

>If one thread could cause an exception being thrown in an other thread,
>would this be considered a polite way to ask? Would it be considered
>an acceptable way?

>>

The exception must not be raised until a point explicitly designed as
safe is hit. Otherwise, any function that manipulates data you'll
still use will potentially be buggered. Consider sys.stdout: codecs,
buffering, lots to go wrong.

>>
>I don't see the point. Exceptions are raised now without the ability
>of an explicitly designed safe point. If something unexpected happens
>your code can raise an exception and leave your data buggered too if
>you didn't anticipate it propely.

>
Although in theory you could get any exception at any point, in
practise you shouldn't unless your program is broken. If it is broken
the exceptions shouldn't be caught and should cause the program to
terminate, so the harm is reduced. The exceptions that should happen
(such as IOError) should be from predicted points, and anticipated.

In pratice you can schedule an alarm and have the alarm handler raise
an exception in your code. It is the resposibility of the developer to
write his code in such a way that it can handle such an interruption
as it should.

Given that the above is already possible I don't see why we should
protect the programmer from such an asynchronous exception merely
because they can be raised from an other thread.

--
Antoon Pardon