generators shared among threads

[jess.austin@gmail.com]

Re: generators shared among threads

Bryan,

You'll get the same result without the lock. I'm not sure what this
indicates. It may show that the contention on the lock and the race
condition on i aren't always problems. It may show that generators, at
least in CPython 2.4, provide thread safety for free. It does seem to
disprove my statement that, "the yield leaves the lock locked".

More than that, I don't know. When threading is involved, different
runs of the same code can yield different results. Can we be sure that
each thread starts where the last one left off? Why wouldn't a thread
just start where it had left off before? Of course, this case would
have the potential for problems that Alex talked about earlier. Why
would a generator object be any more reentrant than a function object?
Because it has a gi_frame attribute? Would generators be thread-safe
only in CPython?

I started the discussion with simpler versions of these same questions.
I'm convinced that using Queue is safe, but now I'm not convinced that
just using a generator is not safe.

cheers,
Jess

[Bryan Olson]

Re: generators shared among threads

jess.austin@gma il.com wrote:[color=blue]
> You'll get the same result without the lock. I'm not sure what this
> indicates. It may show that the contention on the lock and the race
> condition on i aren't always problems. It may show that generators, at
> least in CPython 2.4, provide thread safety for free. It does seem to
> disprove my statement that, "the yield leaves the lock locked".
>
> More than that, I don't know. When threading is involved, different
> runs of the same code can yield different results. Can we be sure that
> each thread starts where the last one left off? Why wouldn't a thread
> just start where it had left off before? Of course, this case would
> have the potential for problems that Alex talked about earlier. Why
> would a generator object be any more reentrant than a function object?
> Because it has a gi_frame attribute? Would generators be thread-safe
> only in CPython?[/color]

I have not found definitive answers in the Python doc. Both
generators and threads keep their own line-of-control, and
how they interact is not clear.


--
--Bryan

[Paul Rubin]

Re: generators shared among threads

Bryan Olson <fakeaddress@no where.org> writes:[color=blue]
> I have not found definitive answers in the Python doc. Both
> generators and threads keep their own line-of-control, and
> how they interact is not clear.[/color]

It looks to me like you can't have two threads in the same generator:

import threading, time

def make_gen():
lock = threading.Lock( )
count = 0
delay = [0,1]
while True:
lock.acquire()

# sleep 1 sec on first iteration, then 0 seconds on next iteration
time.sleep(dela y.pop())
count += 1
yield count

lock.release()

def run():
print gen.next()

gen = make_gen()

# start a thread that will lock the generator for 1 sec
threading.Threa d(target=run).s tart()

# make sure first thread has a chance to get started
time.sleep(0.2)

# start second thread while the generator is locked
threading.Threa d(target=run).s tart()

raises ValueError:

Python 2.3.4 (#1, Feb 2 2005, 12:11:53)
[GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> ## working on region in file /usr/tmp/python-28906xpZ...
>>> Exception in thread Thread-2:[/color][/color][/color]
Traceback (most recent call last):
File "/usr/lib/python2.3/threading.py", line 436, in __bootstrap
self.run()
File "/usr/lib/python2.3/threading.py", line 416, in run
self.__target(* self.__args, **self.__kwargs )
File "/usr/tmp/python-28906xpZ", line 18, in run
ValueError: generator already executing

1

[Tim Peters]

Re: generators shared among threads

[Paul Rubin][color=blue]
> It looks to me like you can't have two threads in the same generator:[/color]

You can't even have one thread in a generator-iterator get away with
activating the generator-iterator while it's already active. That's
an example in PEP 255:

"""
Restriction: A generator cannot be resumed while it is actively
running:
[color=blue][color=green][color=darkred]
>>> def g():[/color][/color][/color]
... i = me.next()
... yield i[color=blue][color=green][color=darkred]
>>> me = g()
>>> me.next()[/color][/color][/color]
Traceback (most recent call last):
...
File "<string>", line 2, in g
ValueError: generator already executing
"""

Same thing if more than one thread tries to do that, but perhaps
harder to see then.

To make some intuitive sense of those, note that a generator-iterator
reuses a single stack frame across resumptions. There is only once
such frame per generator-iterator, hence only (among other things)
one "program counter" per generator-iterator. It should be obvious
that multiple threads can't be allowed to muck with _the_ frame's
program counter simultaneously, and the example above is actually
subtler on that count (because nothing in non-resumable functions
prepares you for that generators make it possible for a _single_
thread to _try_ to "be in two places at the same time" inside a single
invocation of a function -- although any attempt to try to do that
with a single thread is necessarily convoluted, like the example
above, the implementation still has to prevent it).