Not fully understanding the role of Queue.task_done()

[Martin DeMello]

I'm writing a cluster monitor, that collects information from a set of
machines and logs it to a database

In the interests of not hammering the db unnecessarily, I'm
considering the following
1. A series of independent "monitor" threads that collect information
over TCP from the cluster of machines, and write it to a queue
2. A "logger" thread that empties the queue every second or so and
inserts the collected information to the db via a single insert
statement

Reading up on python's built in Queue class, though, it seems oriented
towards "job queues", with a two-step dequeue operation (get() and
task_done()). I'm worried that this would make it too heavyweight for
my application. Is ther documentation somewhere on what exactly
task_done() does, and whether I can disable the tracking of a job once
it's removed from the queue? The python docs for the Queue module were
a bit light.

martin

[Fredrik Lundh]

Re: Not fully understanding the role of Queue.task_done ()

Martin DeMello wrote:

Reading up on python's built in Queue class, though, it seems oriented
towards "job queues", with a two-step dequeue operation (get() and
task_done()). I'm worried that this would make it too heavyweight for
my application. Is ther documentation somewhere on what exactly
task_done() does, and whether I can disable the tracking of a job once
it's removed from the queue? The python docs for the Queue module were
a bit light.

"task_done" just decrements a counter (incremented by "put"). when the
counter reaches zero, the "join" call is unblocked.

</F>

[Martin DeMello]

Re: Not fully understanding the role of Queue.task_done ()

On Sep 4, 12:41 pm, Fredrik Lundh <fred...@python ware.comwrote:

"task_done" just decrements a counter (incremented by "put").  when the
counter reaches zero, the "join" call is unblocked.

Thanks! Is there any standard python idiom to empty a queue into a
list? Or do I just call get() repeatedly and catch the exception when
it's done?

martin

[castironpi]

Re: Not fully understanding the role of Queue.task_done ()

On Sep 4, 2:51 pm, Martin DeMello <martindeme...@ gmail.comwrote:

On Sep 4, 12:41 pm, Fredrik Lundh <fred...@python ware.comwrote:
>

"task_done" just decrements a counter (incremented by "put").  when the
counter reaches zero, the "join" call is unblocked.

>
Thanks! Is there any standard python idiom to empty a queue into a
list? Or do I just call get() repeatedly and catch the exception when
it's done?
>
martin

Random access isn't supported by the defined interface. You can make
it more convenient, though.

import Queue

class IterQueue( Queue.Queue ):
def __iter__( self ):
return self
def next( self ):
if self.empty():
raise StopIteration
return self.get()

q= IterQueue()
q.put( 'a' )
q.put( 'b' )
q.put( 'c' )

print [ x for x in q ]

/Output:
['a', 'b', 'c']

[Martin DeMello]

Re: Not fully understanding the role of Queue.task_done ()

On Sep 4, 1:04 pm, castironpi <castiro...@gma il.comwrote:

>
Random access isn't supported by the defined interface.  You can make
it more convenient, though.

Thanks. I wasn't looking for random access, just wondering what the
cleanest way to implement items = Queue.get_all() was. Your code
should work nicely for that.

martin

[Fredrik Lundh]

Re: Not fully understanding the role of Queue.task_done ()

Martin DeMello wrote:

I'm writing a cluster monitor, that collects information from a set of
machines and logs it to a database
>
In the interests of not hammering the db unnecessarily, I'm
considering the following
1. A series of independent "monitor" threads that collect information
over TCP from the cluster of machines, and write it to a queue
2. A "logger" thread that empties the queue every second or so and
inserts the collected information to the db via a single insert
statement

why are you using a queue for this case, btw? why not just use a plain list

L = []
lock = threading.Lock( )

and add stuff using append in the monitor threads

with lock:
L.append(item)

and regularily reset the list in the logger thread

with lock:
data = L[:]
L[:] = [] # clear the list
for item in data:
... insert into database ...

(list append and assignments to global variables are atomic in CPython,
so you can eliminate the lock by being a bit clever, but that's probably
better left for a non-premature optimization pass).

</F>

[Martin DeMello]

Re: Not fully understanding the role of Queue.task_done ()

On Sep 4, 1:51 pm, Fredrik Lundh <fred...@python ware.comwrote:

Martin DeMello wrote:

I'm writing a cluster monitor, that collects information from a set of
machines and logs it to a database

>

In the interests of not hammering the db unnecessarily, I'm
considering the following
1. A series of independent "monitor" threads that collect information
over TCP from the cluster of machines, and write it to a queue
2. A "logger" thread that empties the queue every second or so and
inserts the collected information to the db via a single insert
statement

>
why are you using a queue for this case, btw?  why not just use a plainlist
>
     L = []
     lock = threading.Lock( )

Good point - I thought of queue because it was self-locking, but
you're right, I can as well use a simple list and lock it myself.

martin

[Aahz]

Re: Not fully understanding the role of Queue.task_done ()

In article <mailman.504.12 20561513.3487.p ython-list@python.org >,
Fredrik Lundh <fredrik@python ware.comwrote:

>Martin DeMello wrote:

>>
>In the interests of not hammering the db unnecessarily, I'm
>considering the following
>1. A series of independent "monitor" threads that collect information
>over TCP from the cluster of machines, and write it to a queue
>2. A "logger" thread that empties the queue every second or so and
>inserts the collected information to the db via a single insert
>statement

>
>why are you using a queue for this case, btw? why not just use a plain list
>
L = []
lock = threading.Lock( )
>
>and add stuff using append in the monitor threads
>
with lock:
L.append(item)

Because using a queue requires less thinking. I certainly would use a
queue in this case instead of rolling my own.
--
Aahz (aahz@pythoncra ft.com) <* http://www.pythoncraft.com/

"Argue for your limitations, and sure enough they're yours." --Richard Bach

[Fredrik Lundh]

Re: Not fully understanding the role of Queue.task_done ()

Aahz wrote:

>why are you using a queue for this case, btw? why not just use a plain list
>>
> L = []
> lock = threading.Lock( )
>>
>and add stuff using append in the monitor threads
>>
> with lock:
> L.append(item)

>
Because using a queue requires less thinking.

given that the whole reason for this thread was that Queue API didn't
fit the OP:s problem, that's a rather dubious statement.

(btw, I've always thought that Python was all about making it easy to
express the solution to a given problem in code, not to let you write
programs without using your brain. when did that change?)

</F>

[alex23]

Re: Not fully understanding the role of Queue.task_done ()

Fredrik Lundh <fred...@python ware.comwrote:

(btw, I've always thought that Python was all about making it easy to
express the solution to a given problem in code, not to let you write
programs without using your brain.  when did that change?)

The day Google App Engine was opened up to developers, I believe.