thread.error: release unlocked lock

**Tim Peters** · Jul 18 '05, 05:48 PM

Re: thread.error: release unlocked lock

[John P.Speno][color=blue]
> So you have this problem, and you decide to use threads to solve it...
>
> System is Python 2.3.3 on Solaris 9.
>
> I'm using the classic Python thread model. Main thread puts stuff into
> a Queue.Queue() and worker threads get stuff out of it and do their
> thing.[/color]

A bounded queue or an unbounded queue? Blocking get/put or
non-blocking get/put? About how many items are in the queue? Since
nobody has reported your symptom before, some detail or other is going
to be important. How do the standard test_thread, test_threading, and
test_queue tests fare on this box? Was this Python configured to use
pthreads or native Solaris threads?
[color=blue]
> On occaision, I get an exception in the main thread when it tries to
> put something into the Queue.
>
> File "/usr/local/bin/poller.py", line 47, in fragnap_it
> work_queue.put( (foo, bar, baz))
> File "/usr/local/lib/python2.3/Queue.py", line 106, in put
> self.fsema.rele ase()
> thread.error: release unlocked lock
>
> Seems like a bad thing, and I'm not sure what can be done. Should I
> catch the thread.error and retry the put()? Or just give up and
> shutdown nicely? (Or quickly finish the twisted version of this code?)
>
> Any idea on the underlying cause of this exception?[/color]

Something has gone insane -- this should be impossible. fsema is a
mutex. put() always acquires fsema. Before it exits, put() releases
fsema again (unless the queue is bounded and put() has just filled
it). The error you're seeing is the mutex complaining about an
attempt to release it when it's not in the acquired state (which
should be impossible because put() acquires fsema before this point).

**John P. Speno** · Jul 18 '05, 05:49 PM

Re: thread.error: release unlocked lock

In <mailman.6568.1 100820402.5135. python-list@python.org > Tim Peters <tim.peters@gma il.com> writes:
[color=blue]
>[John P.Speno][color=green]
>> I'm using the classic Python thread model. Main thread puts stuff into
>> a Queue.Queue() and worker threads get stuff out of it and do their
>> thing.[/color][/color]
[color=blue]
>A bounded queue or an unbounded queue? Blocking get/put or
>non-blocking get/put? About how many items are in the queue? Since[/color]

Thanks for asking. Here's what I can tell you.

Unbounded queues with non-blocking get/put. At most there are 1500 items
in the queue. There are 20 threads getting from the queue, and one
putting.

I did have bounded queues (max 40 items) and blocking puts but got the same
insane error then too.
[color=blue]
>nobody has reported your symptom before, some detail or other is going
>to be important. How do the standard test_thread, test_threading, and
>test_queue tests fare on this box? Was this Python configured to use
>pthreads or native Solaris threads?[/color]

% grep -i thread pyconfig.h | grep -v ^/
#define HAVE_PTHREAD_H 1
#define HAVE_PTHREAD_SI GMASK 1
#define HAVE_THREAD_H 1
#define PTHREAD_SYSTEM_ SCHED_SUPPORTED 1
#define SIZEOF_PTHREAD_ T 4
#define WITH_THREAD 1

I think that means it is using native Solaris threads, assuming my reading
of thread.c is good.

All of test_thread, test_threading, and test_queue all passed.

**Tim Peters** · Jul 18 '05, 05:57 PM

Re: thread.error: release unlocked lock

[John P.Speno][color=blue][color=green][color=darkred]
>>> I'm using the classic Python thread model. Main thread puts stuff
>>> into a Queue.Queue() and worker threads get stuff out of it and do
>>> their thing.[/color][/color][/color]

[Tim Peters][color=blue][color=green]
>> A bounded queue or an unbounded queue? Blocking get/put or
>> non-blocking get/put? About how many items are in the queue?
>> Since nobody has reported your symptom before, some detail or
>> other is going to be important.[/color][/color]

[John][color=blue]
> Thanks for asking. Here's what I can tell you.
>
> Unbounded queues with non-blocking get/put.[/color]

That's odd -- non-blocking put doesn't really make sense with an
unbounded queue, esp. when only one thread does put(). It will work
anyway, it's just peculiar.

So if the queue is unbounded, the put() method is the only code
anywhere that acquires or releases fsema. That simplifies analysis,
but only in that it leads most directly to the conclusion that it's
impossible to get the error you're seeing. The candidates remaining
are things like compiler optimization bugs, buggy platform threads,
buggy C extension modules.

You should really file a bug report, on Python's SourceForge tracker
-- there's already too much detail to keep straight in
email/newsgroup.
[color=blue]
> At most there are 1500 items in the queue. There are 20 threads
> getting from the queue, and one putting.
>
> I did have bounded queues (max 40 items) and blocking puts but got
> the same insane error then too.[/color]

Since nobody else has reported this problem, a successful conclusion
is going to require analyzing a small-as-possible failing test case.
I'm afraid that falls on you, unless someone else can reproduce your
symptom. Here's a start, modeling one plausible set of guesses for
exactly what all the above means:

"""
import threading
import time
from Queue import Queue, Empty
from random import random

class Worker(threadin g.Thread):
def run(self):
while True:
try:
k = q.get_nowait()
assert k == 1
except Empty:
pass
time.sleep(rand om()/10.0)

q = Queue()
ts = [Worker() for dummy in range(20)]
for t in ts:
t.start()

while True:
if q.qsize() > 1500:
print q.qsize()
time.sleep(3)
q.put_nowait(1)
time.sleep(0.00 1)
"""

I can apparently run that all day under 2.3.4 (or 2.4c1, which has an
entirely different Queue implementation) and not see anything odd.
Does it fail for you? If not, how does it differ from what your app
does? IOW, can you fiddle it so that it does fail?
[color=blue][color=green]
>> How do the standard test_thread, test_threading, and
>> test_queue tests fare on this box? Was this Python configured to
>> use pthreads or native Solaris threads?[/color][/color]
[color=blue]
> % grep -i thread pyconfig.h | grep -v ^/
> #define HAVE_PTHREAD_H 1
> #define HAVE_PTHREAD_SI GMASK 1
> #define HAVE_THREAD_H 1
> #define PTHREAD_SYSTEM_ SCHED_SUPPORTED 1
> #define SIZEOF_PTHREAD_ T 4
> #define WITH_THREAD 1
>
> I think that means it is using native Solaris threads, assuming my
> reading of thread.c is good.[/color]

I don't really know about Solaris config (another reason this would be
better in a bug report, where more people who care about bugs <wink>
would pay attention), but I'd *guess* it means you're using pthreads.
[color=blue]
> All of test_thread, test_threading, and test_queue all passed.[/color]

That figures. They check for sanity, but don't really count as stress tests.

**John P. Speno** · Jul 18 '05, 05:57 PM

Re: thread.error: release unlocked lock

In <mailman.6665.1 101018083.5135. python-list@python.org > Tim Peters <tim.peters@gma il.com> writes:
[color=blue]
>[John][color=green]
>> Unbounded queues with non-blocking get/put.[/color][/color]
[color=blue]
>[Tim]
>That's odd -- non-blocking put doesn't really make sense with an
>unbounded queue, esp. when only one thread does put(). It will work
>anyway, it's just peculiar.[/color]

I'll bring this issue into a bug report but I wanted to clarify on
this one issue.

My worker threads used a bare queue.get() with no arguments which does
block. I mistakenly thought the default for the block argument was False
when, in fact, it defaults to True. Sorry about that. I'll modify your
example code accordingly and start testing this asap.

Take care.

thread.error: release unlocked lock

thread.error: release unlocked lock

Comment

Comment

Comment

Comment