How to increase number of threads per process?

[Ronan Viernes]

Hi,

I have created a python script (see below) to count the maximum number
of threads per process (by starting new threads continuously until it
breaks).

######
#testThread.py
import thread, sys

def main():
print "Main Thread:", thread.get_iden t()
count = 0;
try:
while 1:
thread.start_ne w_thread(test,( `count`,))
count = count + 1;
except:
print "Total Threads:", count
print "Exiting Main Thread:", thread.get_iden t()
raise

def test(input=None ):
print "count:", thread.get_iden t(), input
while 1: #keep thread alive until it breaks
pass

if __name__ == "__main__":
main()

#####

Results:

1. SuSE Professional 7.1 (Kernel 2.4.18), Python 1.5.2
Max Threads = 1024

2. SuSE Professional 7.1 (Kernel 2.4.18), Python 2.0
Max Threads = 1024

3. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 1.5.2
Max Threads = 256

4. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 2.2
Max Threads = 512

Note:
For all setup, SuSE Linux threads-max=14336 and max_map_count=6 5536

Questions:
1. How to determine the number of threads? Is it something
configurable?
2. Why do the results above differ in output?
3. How to increase the maximum number of threads per process?


Thanks and hope to hear soon.
Ronan

[Steve]

Re: How to increase number of threads per process?

Ronan,

Ronan Viernes wrote:[color=blue]
> Hi,
>
> I have created a python script (see below) to count the maximum number
> of threads per process (by starting new threads continuously until it
> breaks).
>
> ######
> #testThread.py
> import thread, sys
>
> def main():
> print "Main Thread:", thread.get_iden t()
> count = 0;
> try:
> while 1:
> thread.start_ne w_thread(test,( `count`,))
> count = count + 1;
> except:
> print "Total Threads:", count
> print "Exiting Main Thread:", thread.get_iden t()
> raise
>
> def test(input=None ):
> print "count:", thread.get_iden t(), input
> while 1: #keep thread alive until it breaks
> pass
>
> if __name__ == "__main__":
> main()
>
> #####
>
> Results:
>
> 1. SuSE Professional 7.1 (Kernel 2.4.18), Python 1.5.2
> Max Threads = 1024
>
> 2. SuSE Professional 7.1 (Kernel 2.4.18), Python 2.0
> Max Threads = 1024
>
> 3. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 1.5.2
> Max Threads = 256
>
> 4. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 2.2
> Max Threads = 512
>
> Note:
> For all setup, SuSE Linux threads-max=14336 and max_map_count=6 5536
>
> Questions:
> 1. How to determine the number of threads? Is it something
> configurable?
> 2. Why do the results above differ in output?
> 3. How to increase the maximum number of threads per process?
>[/color]

I'm not sure if this is configurable but I was testing out something
similar today (on fedora core 1). I discovered that a structure such as
the following works with any number of thread (I tried it with 2000
threads just now). The main thing I did was introducing a
'time.sleep(0.0 1)' in the loop that waits for the worker threads to join
the main thread, and adding a try..except clause around where you start
the threads. Basically an exception is never thrown because I run 10
threads at one go, and then sleep for a fraction of a second (it's
usually a good idea to sleep/yield in busy loops). I think the main
thing here is the fact that it takes 10 threads at a time and then gives
it a pause.

Please try this out and let me know what you find.

class A:
cache = {}
counter = 0
def local(self):
A.cache[A.counter*2] = A.counter
A.counter += 1

def print_it(self):
print A.cache, len(A.cache), A.counter


def test():
count = 2000 # I'm pretty sure a larger number should work too

a = A()
threadList = []
for i in xrange(count):
try:
t = threading.Threa d(target=a.loca l, args=())
threadList.appe nd(t)
t.start()

# Adding the following if statement allowed running more
than 256 threads in one go
# Previously it would run only 256 threads if this clause
wasn't there.
if i % 10 == 0 and i != 0:
# setting sys.setcheckint erval to 1 didn't help. Let's try
# voluntarily giving up some cycles -- this should give
# other threads a chance to run
time.sleep(0.01 ) # this works with time.sleep(0.00 5)
as well
# end if
except:
continue
# end try
# end for

# Wait for threads to join the main thread
while threading.activ eCount()>1:
time.sleep(1)

a.print_it()

test()

[color=blue]
>
> Thanks and hope to hear soon.
> Ronan[/color]

Steve

[Ronan Viernes]

Re: How to increase number of threads per process?

Hi Steve,

Thanks for the info. Unfortunately, adding the sleep to my previous
example (modified version below) did not alter the results. Please
take note that I am using thread library (and not threading). The
sample code below is simplified version of our system and of course
each of the thread has other tasks to do rather than staying idle in
there.

What really puzzles me now is, if I change the environment (OS and
Python Versions) accordingly it will give different results. This made
me believe that it is somewhat 'configurable'.

Anyway, I really appreciate the help. Let's wait what the others have
to say.
[color=blue][color=green]
> > ######
> > #testThread.py
> > import thread, sys
> >
> > def main():
> > print "Main Thread:", thread.get_iden t()
> > count = 0;
> > try:
> > while 1:
> > thread.start_ne w_thread(test,( `count`,))
> > count = count + 1;
> > if count % 10 == 0 and count != 0:
> > time.sleep(0.01 )
> > except:
> > print "Total Threads:", count
> > print "Exiting Main Thread:", thread.get_iden t()
> > raise
> >
> > def test(input=None ):
> > print "count:", thread.get_iden t(), input
> > while 1: #keep thread alive until it breaks
> > pass
> >
> > if __name__ == "__main__":
> > main()
> >
> > #####
> >
> > Results:
> >
> > 1. SuSE Professional 7.1 (Kernel 2.4.18), Python 1.5.2
> > Max Threads = 1024
> >
> > 2. SuSE Professional 7.1 (Kernel 2.4.18), Python 2.0
> > Max Threads = 1024
> >
> > 3. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 1.5.2
> > Max Threads = 256
> >
> > 4. SuSE Enterprise Server 8.0 (Kernel 2.4.18), Python 2.2
> > Max Threads = 512
> >
> > Note:
> > For all setup, SuSE Linux threads-max=14336 and max_map_count=6 5536
> >
> > Questions:
> > 1. How to determine the number of threads? Is it something
> > configurable?
> > 2. Why do the results above differ in output?
> > 3. How to increase the maximum number of threads per process?
> >[/color][/color]

Regards,
Ronan

[John Lenton]

Re: How to increase number of threads per process?

On 13 Jul 2004 19:27:21 -0700, Ronan Viernes <ronan.viernes@ onsemi.com> wrote:[color=blue]
> Hi Steve,
>
> Thanks for the info. Unfortunately, adding the sleep to my previous
> example (modified version below) did not alter the results. Please
> take note that I am using thread library (and not threading). The
> sample code below is simplified version of our system and of course
> each of the thread has other tasks to do rather than staying idle in
> there.
>
> What really puzzles me now is, if I change the environment (OS and
> Python Versions) accordingly it will give different results. This made
> me believe that it is somewhat 'configurable'.[/color]

as far as I know, it's the way your glibc is compiled that makes the difference.

--
John Lenton (jlenton@gmail. com) -- Random fortune:
bash: fortune: command not found