Good thread pool module

[David Hirschfield]

There isn't a thread pool module in the standard library, but I'm sure
many have been written by people in the python community.
Anyone have a favorite? Is there one particular implementation that's
recommended?

Not looking for anything fancy, just something that lets me queue up
tasks to be performed by a pool of threads and then retrieve results
when the tasks complete.

Thanks,
-Dave

--
Presenting:
mediocre nebula.

[Rune Hansen]

Re: Good thread pool module

I think you want the Queue module in standard lib. Haven't started a
thread yet without it :)

regards
/rune

[Steve M]

Re: Good thread pool module

I believe "Python in a Nutshell" has a couple of clear examples using
Queue and Threading, including one with a pool of worker threads that
wait for entries in one queue and place results in another.

Also you should look at the Python Cookbook, which probably includes
the same or similar examples from the Nutshell book, since the author
of that is an editor of Cookbook.

http://aspn.activestate.com/ASPN/Cookbook/Python?kwd=Threads

[Raymond Hettinger]

Re: Good thread pool module

David Hirschfield wrote:[color=blue]
> There isn't a thread pool module in the standard library, but I'm sure
> many have been written by people in the python community.
> Anyone have a favorite? Is there one particular implementation that's
> recommended?[/color]

Because of the GIL, thread pools are not as useful in Python as you
might expect -- they execute one at a time and do not take advantage of
hyper-threading or multiple processors. If that kind of efficiency is
what you are after, then take a look at PyLinda which can coordinate
communication between multiple instances of Python running in separate
threads:

http://www-users.cs.york.ac.uk/~aw/pylinda/

[color=blue]
> Not looking for anything fancy, just something that lets me queue up
> tasks to be performed by a pool of threads and then retrieve results
> when the tasks complete.[/color]

FWIW, I wrote a small enchancement to the Queue module that makes it a
little bit easier to use a pool of worker threads and then be able to
tell when they are all done with processing:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/475160

Raymond

[Rene Pijlman]

Re: Good thread pool module

Dennis Lee Bieber:[color=blue]
>Raymond Hettinger:[color=green]
>> Because of the GIL, thread pools are not as useful in Python as you
>> might expect -- they execute one at a time and do not take advantage of
>> hyper-threading or multiple processors. If that kind of efficiency is[/color]
>
> If the task is I/O bound (something like a web spider?), seems
>they'd still be useful...[/color]

Yes, I use a thread pool for that. But async I/O may be a more efficient
solution (e.g. Twisted).

--
René Pijlman

Wat wil jij leren? http://www.leren.nl