multiprocessing: queue.get() blocks even if queue.qsize() != 0

[redbaron]

I run into problem with queue from multiprocessing . Even if I
queue.qsize() != 0 queue.get() still blocks and queue.get_nowai t()
raises Emtpy error.

I'm unable to cut my big part to small test case, because smaller test
case similair to my real app by design is works. In what conditions is
it possible?

while qresult.qsize() :
result = qresult.get() #this code blocks!
doWithResult(re sult)

[MRAB]

Re: multiprocessing : queue.get() blocks even if queue.qsize() != 0

On Oct 15, 2:05 pm, redbaron <ivanov.ma...@g mail.comwrote:

I run into problem with queue from multiprocessing . Even if I
queue.qsize() != 0 queue.get() still blocks and queue.get_nowai t()
raises Emtpy error.
>
I'm unable to cut my big part to small test case, because smaller test
case similair to my real app by design is works. In what conditions is
it possible?
>
while qresult.qsize() :
    result = qresult.get()  #this code blocks!
    doWithResult(re sult)

From Python v2.5 onwards queues also have a task_done() method. Try:

while qresult.qsize() :
result = qresult.get() #this code blocks!
doWithResult(re sult)
qresult.task_do ne()

[Antoon Pardon]

Re: multiprocessing : queue.get() blocks even if queue.qsize() != 0

On 2008-10-15, redbaron <ivanov.maxim@g mail.comwrote:

I run into problem with queue from multiprocessing . Even if I
queue.qsize() != 0 queue.get() still blocks and queue.get_nowai t()
raises Emtpy error.
>
I'm unable to cut my big part to small test case, because smaller test
case similair to my real app by design is works. In what conditions is
it possible?
>
while qresult.qsize() :
result = qresult.get() #this code blocks!
doWithResult(re sult)

If you have more than one consumer the above code can block.
The two consumers both see that there is an item present
in the queue. One removes the item and the second blocks.

--
Antoon Pardon

[Paul Rubin]

Re: multiprocessing : queue.get() blocks even if queue.qsize() != 0

redbaron <ivanov.maxim@g mail.comwrites:

while qresult.qsize() :
result = qresult.get() #this code blocks!
doWithResult(re sult)

That is unreliable for the reason Antoon explained, and as is
documented in the manual for the Queue module. Write instead
something like (untested):

while True:
try:
result = qresult.get_now ait()
except Empty:
break
doWithResult(re sult)