Using multiprocessing

[nhwarriors]

I am attempting to use the (new in 2.6) multiprocessing package to
process 2 items in a large queue of items simultaneously. I'd like to
be able to print to the screen the results of each item before
starting the next one. I'm having trouble with this so far.

Here is some (useless) example code that shows how far I've gotten by
reading the documentation:

from multiprocessing import Process, Queue, current_process

def main():
facs = []
for i in range(50000,500 05):
facs.append(i)

tasks = [(fac, (i,)) for i in facs]
task_queue = Queue()
done_queue = Queue()

for task in tasks:
task_queue.put( task)

for i in range(2):
Process(target = worker, args = (task_queue, done_queue)).st art()

for i in range(len(tasks )):
print done_queue.get( )

for i in range(2):
task_queue.put( 'STOP')

def worker(input, output):
for func, args in iter(input.get, 'STOP'):
result = func(*args)
output.put(resu lt)

def fac(n):
f = n
for i in range(n-1,1,-1):
f *= i
return 'fac('+str(n)+' ) done on '+current_proce ss().name

if __name__ == '__main__':
main()

This works great, except that nothing can be output until everything
in the queue is finished. I'd like to write out the result of fac(n)
for each item in the queue as it happens.

I'm probably approaching the problem all wrong - can anyone set me on
the right track?

[Jesse Noller]

Re: Using multiprocessing

On Fri, Oct 10, 2008 at 4:32 PM, nhwarriors <edward.reed@gm ail.comwrote:

I am attempting to use the (new in 2.6) multiprocessing package to
process 2 items in a large queue of items simultaneously. I'd like to
be able to print to the screen the results of each item before
starting the next one. I'm having trouble with this so far.
>
Here is some (useless) example code that shows how far I've gotten by
reading the documentation:
>
from multiprocessing import Process, Queue, current_process
>
def main():
facs = []
for i in range(50000,500 05):
facs.append(i)
>
tasks = [(fac, (i,)) for i in facs]
task_queue = Queue()
done_queue = Queue()
>
for task in tasks:
task_queue.put( task)
>
for i in range(2):
Process(target = worker, args = (task_queue, done_queue)).st art()
>
for i in range(len(tasks )):
print done_queue.get( )
>
for i in range(2):
task_queue.put( 'STOP')
>
def worker(input, output):
for func, args in iter(input.get, 'STOP'):
result = func(*args)
output.put(resu lt)
>
def fac(n):
f = n
for i in range(n-1,1,-1):
f *= i
return 'fac('+str(n)+' ) done on '+current_proce ss().name
>
if __name__ == '__main__':
main()
>
This works great, except that nothing can be output until everything
in the queue is finished. I'd like to write out the result of fac(n)
for each item in the queue as it happens.
>
I'm probably approaching the problem all wrong - can anyone set me on
the right track?

I'm not quite following: If you run this, the results are printed by
the main thread, unordered, as they are put on the results queue -
this works as intended (and the example this is based on works the
same way) .

For example:
result put Process-2
result put Process-1
fac(50000) done on Process-1
result put Process-2
fac(50001) done on Process-2
result put Process-1
fac(50003) done on Process-1
result put Process-2
fac(50002) done on Process-2
fac(50004) done on Process-2

You can see this if you expand the range:

result put Process-1
result put Process-2
result put Process-2
fac(50001) done on Process-2
result put Process-1
fac(50000) done on Process-1
fac(50003) done on Process-2
result put Process-2
result put Process-1
fac(50004) done on Process-2
result put Process-2
fac(50006) done on Process-2
result put Process-1
fac(50002) done on Process-1
fac(50005) done on Process-1
fac(50007) done on Process-1
result put Process-2
result put Process-1
fac(50008) done on Process-2
result put Process-2
result put Process-1
fac(50010) done on Process-2
result put Process-2
result put Process-1
fac(50009) done on Process-1
fac(50011) done on Process-1
fac(50013) done on Process-1
result put Process-2
fac(50012) done on Process-2
fac(50014) done on Process-2

One trick I use is when I have a results queue to manage, I spawn an
addition process to read off of the results queue and deal with the
results. This is mainly so I can process the results outside of the
main thread, as they appear on the results queue

-jesse

[Aaron \Castironpi\ Brady]

Re: Using multiprocessing

On Oct 10, 3:32 pm, nhwarriors <edward.r...@gm ail.comwrote:

I am attempting to use the (new in 2.6) multiprocessing package to
process 2 items in a large queue of items simultaneously. I'd like to
be able to print to the screen the results of each item before
starting the next one. I'm having trouble with this so far.
>
Here is some (useless) example code that shows how far I've gotten by
reading the documentation:
>

snip code

>
This works great, except that nothing can be output until everything
in the queue is finished. I'd like to write out the result of fac(n)
for each item in the queue as it happens.
>
I'm probably approaching the problem all wrong - can anyone set me on
the right track?

Works fine for me. Formatting time.clock(), with a different range:

0.00000 fac(25000) done on Process-2
0.09334 fac(25001) done on Process-1
2.25036 fac(25002) done on Process-2
2.41227 fac(25003) done on Process-1
3.57167 fac(25004) done on Process-2

I'm on Win32.

[nhwarriors]

Re: Using multiprocessing

On Oct 10, 10:52 pm, "Aaron \"Castironpi \" Brady"
<castiro...@gma il.comwrote:

On Oct 10, 3:32 pm, nhwarriors <edward.r...@gm ail.comwrote:
>
>
>

I am attempting to use the (new in 2.6) multiprocessing package to
process 2 items in a large queue of items simultaneously. I'd like to
be able to print to the screen the results of each item before
starting the next one. I'm having trouble with this so far.

>

Here is some (useless) example code that shows how far I've gotten by
reading the documentation:

>
snip code
>

This works great, except that nothing can be output until everything
in the queue is finished. I'd like to write out the result of fac(n)
for each item in the queue as it happens.

>

I'm probably approaching the problem all wrong - can anyone set me on
the right track?

>
Works fine for me.  Formatting time.clock(), with a different range:
>
0.00000 fac(25000) done on Process-2
0.09334 fac(25001) done on Process-1
2.25036 fac(25002) done on Process-2
2.41227 fac(25003) done on Process-1
3.57167 fac(25004) done on Process-2
>
I'm on Win32.

Strange. I was on Win32 (Cygwin) earlier today and it was doing what I
reported above. At home on Linux, it works as I wanted and you
experienced. Must be something screwy with running it in Cygwin.

Thanks guys!

[Aaron \Castironpi\ Brady]

Re: Using multiprocessing

On Oct 10, 10:48 pm, nhwarriors <edward.r...@gm ail.comwrote:

On Oct 10, 10:52 pm, "Aaron \"Castironpi \" Brady"
>
>
>
<castiro...@gma il.comwrote:

On Oct 10, 3:32 pm, nhwarriors <edward.r...@gm ail.comwrote:

>

I am attempting to use the (new in 2.6) multiprocessing package to
process 2 items in a large queue of items simultaneously. I'd like to
be able to print to the screen the results of each item before
starting the next one. I'm having trouble with this so far.

>

Here is some (useless) example code that shows how far I've gotten by
reading the documentation:

>

snip code

>

This works great, except that nothing can be output until everything
in the queue is finished. I'd like to write out the result of fac(n)
for each item in the queue as it happens.

>

I'm probably approaching the problem all wrong - can anyone set me on
the right track?

>

Works fine for me.  Formatting time.clock(), with a different range:

>

0.00000 fac(25000) done on Process-2
0.09334 fac(25001) done on Process-1
2.25036 fac(25002) done on Process-2
2.41227 fac(25003) done on Process-1
3.57167 fac(25004) done on Process-2

>

I'm on Win32.

>
Strange. I was on Win32 (Cygwin) earlier today and it was doing what I
reported above. At home on Linux, it works as I wanted and you
experienced. Must be something screwy with running it in Cygwin.
>
Thanks guys!

You might need to flush the standard out between calls. That gets
suggested from time to time. Not sure what Python's req.s are.