How to force a thread to stop

[Hans]

Hi all,

Is there a way that the program that created and started a thread also stops
it.
(My usage is a time-out).

E.g.

thread = threading.Threa d(target=Loop.t estLoop)
thread.start() # This thread is expected to finish within a second
thread.join(2) # Or time.sleep(2) ?

if thread.isAlive( ):
# thread has probably encountered a problem and hangs
# What should be here to stop thread ??????

Note that I don't want to change the target (too much), as many possible
targets exist,
together thousands of lines of code.

Thanks,
Hans

[bryanjugglercryptographer@yahoo.com]

Re: How to force a thread to stop


Hans wrote:

Hi all,
>
Is there a way that the program that created and started a thread also stops
it.
(My usage is a time-out).
>
E.g.
>
thread = threading.Threa d(target=Loop.t estLoop)
thread.start() # This thread is expected to finish within a second
thread.join(2) # Or time.sleep(2) ?

No, Python has no threadicide method, and its absence is not an
oversight. Threads often have important business left to do, such
as releasing locks on shared data; killing them at arbitrary times
tends to leave the system in an inconsistent state.

if thread.isAlive( ):
# thread has probably encountered a problem and hangs
# What should be here to stop thread ??????

At this point, it's too late. Try to code so your threads don't hang.

Python does let you arrange for threads to die when you want to
terminate the program, with threading's Thread.setDaemo n().


--
--Bryan

[Méta-MCI]

Re: How to force a thread to stop

Hi!

>> "threadicid e" method

I like this word...

Michel Claveau

[Lawrence D'Oliveiro]

Re: How to force a thread to stop

In message <1153582976.746 036.312330@m73g 2000cwd.googleg roups.com>,
bryanjugglercry ptographer@yaho o.com wrote:

Python has no threadicide method, and its absence is not an
oversight. Threads often have important business left to do, such
as releasing locks on shared data; killing them at arbitrary times
tends to leave the system in an inconsistent state.

Perhaps another reason to avoid threads and use processes instead?

[Paul Rubin]

Re: How to force a thread to stop

Lawrence D'Oliveiro <ldo@geek-central.gen.new _zealandwrites:

Python has no threadicide method, and its absence is not an
oversight. Threads often have important business left to do, such
as releasing locks on shared data; killing them at arbitrary times
tends to leave the system in an inconsistent state.

>
Perhaps another reason to avoid threads and use processes instead?

If the processes are sharing resources, the exact same problems arise.

[Carl J. Van Arsdall]

Re: How to force a thread to stop

Dennis Lee Bieber wrote:

On Sat, 22 Jul 2006 14:47:30 +0200, "Hans" <NoSpam@Hccnet. nldeclaimed
the following in comp.lang.pytho n:
>
>

>Hi all,
>>
>Is there a way that the program that created and started a thread also stops
>it.
>(My usage is a time-out).
>>
>>

Hasn't this subject become a FAQ entry yet? <G>
>
The only reliable way of stopping a thread is from inside the thread
itself. That is, the thread must, at some point, examine some flag
variable which, when set, says "stop"
>
Without using actual code:
>
class StoppableThread (...):
def __init__(self, ...):
#whatever is needed to initialize as a thread
self.Stop = False
>
def run(self, ...):
while not self.Stop:
#do one cycle of the computation
>
>
>

My problem with the fact that python doesn't have some type of "thread
killer" is that again, the only solution involves some type of polling
loop. I.e. "if your thread of execution can be written so that it
periodically checks for a kill condition". This really sucks, not just
because polling is a ridiculous practice, but it forces programmers in
many situations to go through a lengthy process of organizing operations
into a list. For, say I have threads that share a bunch of common
memory (yea, i'm saying this exclusively to get the procses users off my
back) that executes a series of commands on remote nodes using rsh or
something. So if i've constructed my system using threads I need to
neatly go and dump all operations into some sort of list so that I can
implement a polling mechanism, i.e.

opList = [op1, op2, op3, op4]
for op in opList:
checkMessageQue ue()
op()

That works if you can easily create an opList. If you want good
response time this can become quite ugly, especially if you have a lot
going on. Say I have a function I want to run in a thread:

#Just pretend for the sake of arguement that 'op' actually means
something and is a lengthy operation
def func_to_thread( ):
os.system('op 1')
os.system('op 2')
os.system('op 3')


#In order to make this killable with reasonable response time we have to
organize each of our ops into a function or something equally annoying

op_1():
os.system('op 1')

op_2():
os.system('op 2')

op_3():
os.system('op 3')

opList(op_1, op_2, op_3)
def to_thread():
for op in opList:
checkMessageQue ue()
op()


So with this whole "hey mr. nice thread, please die for me" concept gets
ugly quickly in complex situations and doesn't scale well at all.
Furthermore, say you have a complex systems where users can write
pluggable modules. IF a module gets stuck inside of some screwed up
loop and is unable to poll for messages there's no way to kill the
module without killing the whole system. Any of you guys thought of a
way around this scenario?


--

Carl J. Van Arsdall
cvanarsdall@mvi sta.com
Build and Release
MontaVista Software

[Steve Holden]

Re: How to force a thread to stop

Carl J. Van Arsdall wrote:
[... rant ...]

So with this whole "hey mr. nice thread, please die for me" concept gets
ugly quickly in complex situations and doesn't scale well at all.
Furthermore, say you have a complex systems where users can write
pluggable modules. IF a module gets stuck inside of some screwed up
loop and is unable to poll for messages there's no way to kill the
module without killing the whole system. Any of you guys thought of a
way around this scenario?
>
>

Communications through Queue.Queue objects can help. But if you research
the history of this design decision in the language you should discover
there are fairly sound rasons for not allowing arbitrary "threadicid e".

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Carl J. Van Arsdall]

Re: How to force a thread to stop

Steve Holden wrote:

Carl J. Van Arsdall wrote:
[... rant ...]
>

>So with this whole "hey mr. nice thread, please die for me" concept gets
>ugly quickly in complex situations and doesn't scale well at all.
>Furthermore, say you have a complex systems where users can write
>pluggable modules. IF a module gets stuck inside of some screwed up
>loop and is unable to poll for messages there's no way to kill the
>module without killing the whole system. Any of you guys thought of a
>way around this scenario?
>>
>>
>>

>
Communications through Queue.Queue objects can help. But if you research
the history of this design decision in the language you should discover
there are fairly sound rasons for not allowing arbitrary "threadicid e".
>
>
>

Right, I'm wondering if there was a way to make an interrupt driven
communication mechanism for threads? Example: thread receives a
message, stops everything, and processes the message.


--

Carl J. Van Arsdall
cvanarsdall@mvi sta.com
Build and Release
MontaVista Software

[Paul Rubin]

Re: How to force a thread to stop

"Carl J. Van Arsdall" <cvanarsdall@mv ista.comwrites:

Communications through Queue.Queue objects can help. But if you
research the history of this design decision in the language you
should discover there are fairly sound rasons for not allowing
arbitrary "threadicid e".

Right, I'm wondering if there was a way to make an interrupt driven
communication mechanism for threads? Example: thread receives a
message, stops everything, and processes the message. --

There is in fact some under-the-covers mechanism in CPython (i.e.
one you can call from C extensions but not from Python code) to
raise exceptions in other threads. I've forgotten the details.
There has been discussion at various times about how to expose
something like that to Python code, but it's been inconclusive. E.g.:

http://sf.net/tracker/?func=detail&atid=105470&aid=502236&group_id=5470

[bryanjugglercryptographer@yahoo.com]

Re: How to force a thread to stop


Carl J. Van Arsdall wrote:
[...]

My problem with the fact that python doesn't have some type of "thread
killer" is that again, the only solution involves some type of polling
loop.

A polliing loop is neither required nor helpful here.

[...]

#Just pretend for the sake of arguement that 'op' actually means
something and is a lengthy operation
def func_to_thread( ):
os.system('op 1')
os.system('op 2')
os.system('op 3')

What good do you think killing that thread would do? The
process running 'op n' has no particular binding to the thread
that called os.system(). If 'op n' hangs, it stays hung.

The problem here is that os.system doesn't give you enough
control. It doesn't have a timeout and doesn't give you a
process ID or handle to the spawned process.

Running os.system() in multiple threads strikes me as
kind of whacked. Won't they all compete to read and write
stdin/stdout simultaneously?

#In order to make this killable with reasonable response time we have to
organize each of our ops into a function or something equally annoying
>
op_1():
os.system('op 1')
>
op_2():
os.system('op 2')
>
op_3():
os.system('op 3')
>
opList(op_1, op_2, op_3)
def to_thread():
for op in opList:
checkMessageQue ue()
op()

Nonsense. If op() hangs, you never get to checkMessageQue ue().

Now suppose op has a timeout. We could write

def opcheck(thing):
result = op(thing)
if result == there_was_a_tim eout:
raise some_timeout_ex ception

How is:

def func_to_thread( ):
opcheck('op 1')
opcheck('op 2')
opcheck('op 3')

any less managable than your version of func_to_thread?

So with this whole "hey mr. nice thread, please die for me" concept gets
ugly quickly in complex situations and doesn't scale well at all.
Furthermore, say you have a complex systems where users can write
pluggable modules. IF a module gets stuck inside of some screwed up
loop and is unable to poll for messages there's no way to kill the
module without killing the whole system. Any of you guys thought of a
way around this scenario?

Threadicide would not solve the problems you actually have, and it
tends to create other problems. What is the condition that makes
you want to kill the thread? Make the victim thread respond to that
condition itself.


--
--Bryan

[Paul Rubin]

Re: How to force a thread to stop

bryanjugglercry ptographer@yaho o.com writes:

Threadicide would not solve the problems you actually have, and it
tends to create other problems. What is the condition that makes
you want to kill the thread? Make the victim thread respond to that
condition itself.

If the condition is a timeout, one way to notice it is with sigalarm,
which raises an exception in the main thread. But then you need a way
to make something happen in the remote thread.

[bryanjugglercryptographer@yahoo.com]

Re: How to force a thread to stop


Dennis Lee Bieber wrote:

On Mon, 24 Jul 2006 10:27:08 -0700, "Carl J. Van Arsdall"

My problem with the fact that python doesn't have some type of "thread
killer" is that again, the only solution involves some type of polling
loop. I.e. "if your thread of execution can be written so that it

>
And that is because the control of a thread, once started, is
dependent upon the underlying OS...

No; it's because killing a thread from another thread fundamentally
sloppy.

The process of creating a thread can
be translated into something supplied by pretty much all operating
systems: an Amiga task, posix thread, etc.
>
But ending a thread is then also dependent upon the OS -- and not
all OSs have a way to do that that doesn't run the risk of leaking
memory, leaving things locked, etc. until the next reboot.

No operating system has a good way to do it, at least not for
the kind of threads Python offers.

The procedure for M$ Windows to end a task basically comes down to
"send the task a 'close window' event; if that doesn't work, escalate...
until in the end it throw its hands up and says -- go ahead and leave
memory in a mess, just stop running that thread".

The right procedure in MS Windows is the same as under POSIX:
let the thread terminate on its own.

module without killing the whole system. Any of you guys thought of a
way around this scenario?

>
Ask Bill Gates... The problem is part of the OS.

Or learn how to use threads properly. Linux is starting to get good
threading. Win32 has had it for quite a while.


--
--Bryan

[Carl J. Van Arsdall]

Re: How to force a thread to stop

bryanjugglercry ptographer@yaho o.com wrote:

Carl J. Van Arsdall wrote:
[...]
>

>My problem with the fact that python doesn't have some type of "thread
>killer" is that again, the only solution involves some type of polling
>loop.
>>

>
A polliing loop is neither required nor helpful here.
>
[...]
>

>#Just pretend for the sake of arguement that 'op' actually means
>something and is a lengthy operation
>def func_to_thread( ):
> os.system('op 1')
> os.system('op 2')
> os.system('op 3')
>>

>
What good do you think killing that thread would do? The
process running 'op n' has no particular binding to the thread
that called os.system(). If 'op n' hangs, it stays hung.
>
The problem here is that os.system doesn't give you enough
control. It doesn't have a timeout and doesn't give you a
process ID or handle to the spawned process.
>
Running os.system() in multiple threads strikes me as
kind of whacked. Won't they all compete to read and write
stdin/stdout simultaneously?
>

Unfortunately this is due to the nature of the problem I am tasked with
solving. I have a large computing farm, these os.system calls are often
things like ssh that do work on locations remote from the initial python
task. I suppose eventually I'll end up using a framework like twisted
but, as with many projects, I got thrown into this thing and threading
is where we ended up. So now there's the rush to make things work
before we can really look at a proper solution.

>

>#In order to make this killable with reasonable response time we have to
>organize each of our ops into a function or something equally annoying
>>
>op_1():
> os.system('op 1')
>>
>op_2():
> os.system('op 2')
>>
>op_3():
> os.system('op 3')
>>
>opList(op_1, op_2, op_3)
>def to_thread():
> for op in opList:
> checkMessageQue ue()
> op()
>>

>
Nonsense. If op() hangs, you never get to checkMessageQue ue().
>

Yea, understood. At the same time, I can't use a timeout either, I
don't know how long op_1 or op_2 will be. This is why I want something
that is triggered on an event.

Now suppose op has a timeout. We could write
>
def opcheck(thing):
result = op(thing)
if result == there_was_a_tim eout:
raise some_timeout_ex ception
>
How is:
>
def func_to_thread( ):
opcheck('op 1')
opcheck('op 2')
opcheck('op 3')
>
any less managable than your version of func_to_thread?
>
>

Again, the problem I'm trying to solve doesn't work like this. I've
been working on a framework to be run across a large number of
distributed nodes (here's where you throw out the "duh, use a
distributed technology" in my face). The thing is, I'm only writing the
framework, the framework will work with modules, lots of them, which
will be written by other people. Its going to be impossible to get
people to write hundreds of modules that constantly check for status
messages. So, if I want my thread to "give itself up" I have to tell it
to give up. In order to tell it to give up I need some mechanism to
check messages that is not going to piss off a large team of
programmers. At the same time, do I really want to rely on other people
to make things work? Not really, I'd much rather let my framework
handle all control and not leave that up to programmers.

So the problem is, I have something linearly executed a large list of
python functions of various sizes ranging from short to long. Its not
about killing the thread so much as how do I make the thread listen to
control messages without polling.

>So with this whole "hey mr. nice thread, please die for me" concept gets
>ugly quickly in complex situations and doesn't scale well at all.
>Furthermore, say you have a complex systems where users can write
>pluggable modules. IF a module gets stuck inside of some screwed up
>loop and is unable to poll for messages there's no way to kill the
>module without killing the whole system. Any of you guys thought of a
>way around this scenario?
>>

>
Threadicide would not solve the problems you actually have, and it
tends to create other problems. What is the condition that makes
you want to kill the thread? Make the victim thread respond to that
condition itself.
>
>

I feel like this is something we've established multiple times. Yes, we
want the thread to kill itself. Alright, now that we agree on that,
what is the best way to do that. Right now people keep saying we must
send the thread a message. That's fine and I completely understand
that, but right now the only mechanism I see is some type of polling
loop (or diving into the C API to force exceptions). So far I've not
seen any other method though. If you want to send a thread a control
message you must wait until that thread is able to check for a control
message. If something hangs in your thread you are totally screwed,
similarly, if your thread ends up in some excessively lengthy IO (IO
that could be interrupted or whatever) you have to wait for that IO to
finish before your thread can process any control messages.




--

Carl J. Van Arsdall
cvanarsdall@mvi sta.com
Build and Release
MontaVista Software

[Gerhard Fiedler]

Re: How to force a thread to stop

On 2006-07-25 13:30:22, Carl J. Van Arsdall wrote:

>Running os.system() in multiple threads strikes me as kind of whacked.
>Won't they all compete to read and write stdin/stdout simultaneously?
>>

Unfortunately this is due to the nature of the problem I am tasked with
solving. I have a large computing farm, these os.system calls are often
things like ssh that do work on locations remote from the initial python
task.

[...]

Again, the problem I'm trying to solve doesn't work like this. I've been
working on a framework to be run across a large number of distributed
nodes (here's where you throw out the "duh, use a distributed
technology" in my face). The thing is, I'm only writing the framework,
the framework will work with modules, lots of them, which will be
written by other people. Its going to be impossible to get people to
write hundreds of modules that constantly check for status messages.

Doesn't this sound like a case for using processes instead of threads?
Where you don't have control over the thread, you can use a process and get
the separation you need to be able to kill this task.

Alternatively you could possibly provide a base class for the threads that
handles the things you need every thread to handle. They'd not have to
write it then; they'd not even have to know too much about it.

Gerhard

[Carl J. Van Arsdall]

Re: How to force a thread to stop

Gerhard Fiedler wrote:

On 2006-07-25 13:30:22, Carl J. Van Arsdall wrote:
>
>

>>Running os.system() in multiple threads strikes me as kind of whacked.
>>Won't they all compete to read and write stdin/stdout simultaneously?
>>>
>>>

>Unfortunatel y this is due to the nature of the problem I am tasked with
>solving. I have a large computing farm, these os.system calls are often
>things like ssh that do work on locations remote from the initial python
>task.
>>

>
[...]
>
>

>Again, the problem I'm trying to solve doesn't work like this. I've been
>working on a framework to be run across a large number of distributed
>nodes (here's where you throw out the "duh, use a distributed
>technology" in my face). The thing is, I'm only writing the framework,
>the framework will work with modules, lots of them, which will be
>written by other people. Its going to be impossible to get people to
>write hundreds of modules that constantly check for status messages.
>>

>
Doesn't this sound like a case for using processes instead of threads?
Where you don't have control over the thread, you can use a process and get
the separation you need to be able to kill this task.
>
Alternatively you could possibly provide a base class for the threads that
handles the things you need every thread to handle. They'd not have to
write it then; they'd not even have to know too much about it.
>
Gerhard
>
>

I'd be all for using processes but setting up communication between
processes would be difficult wouldn't it? I mean, threads have shared
memory so making sure all threads know the current system state is an
easy thing. With processes wouldn't I have to setup some type of
server/client design, where one process has the system state and then
the other processes constantly probe the host when they need the current
system state?




--

Carl J. Van Arsdall
cvanarsdall@mvi sta.com
Build and Release
MontaVista Software