Python threading, and processes

[Robin Haswell]

Hey there

I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
Debian machines, and I've noticed that if I open a lot of threads (say,
50), I get lots of python processes with individual PIDs, which consume a
disproportionat e amount of CPU. Does this mean that Python is using the
dummy_thread module by accident? And is there a realistic limitation to
the number of threads I can do?

Cheers

-Rob

[Diez B. Roggisch]

Re: Python threading, and processes

Robin Haswell wrote:
[color=blue]
> Hey there
>
> I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
> Debian machines, and I've noticed that if I open a lot of threads (say,
> 50), I get lots of python processes with individual PIDs, which consume a
> disproportionat e amount of CPU. Does this mean that Python is using the
> dummy_thread module by accident? And is there a realistic limitation to
> the number of threads I can do?[/color]

Both system-depend. The way threads are shwon depends on the systenm. And a
thread that runs, runs - so it consumes cpu-cycles. Try calling
time.sleep() to see how the consumption decreases.

And the limitation of allowed threads per process and/or system is also a
OS-dependend thing.

"""
Under Linux, threads are counted as processes, so any limits to the number
of processes also applies to threads. In a heavily threaded app like a
threaded TCP engine, or a java server, you can quickly run out of threads.
"""

from

http://people.redhat.com/alikins/system_tuning.html#threads

Diez

[Robin Haswell]

Re: Python threading, and processes

On Wed, 08 Feb 2006 14:24:38 +0100, Diez B. Roggisch wrote:
[color=blue]
> Robin Haswell wrote:
>[color=green]
>> Hey there
>>
>> I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
>> Debian machines, and I've noticed that if I open a lot of threads (say,
>> 50), I get lots of python processes with individual PIDs, which consume a
>> disproportionat e amount of CPU. Does this mean that Python is using the
>> dummy_thread module by accident? And is there a realistic limitation to
>> the number of threads I can do?[/color]
>
> Both system-depend. The way threads are shwon depends on the systenm. And a
> thread that runs, runs - so it consumes cpu-cycles. Try calling
> time.sleep() to see how the consumption decreases.[/color]

Cheers for that info. The thread's main tasks are getting webpages
(spidering), the actual amount of processing done in each thread is
minimal - that's why I'm confused by the CPU usage.

Cheers

-Rob
[color=blue]
>
> Diez[/color]

[Alex Martelli]

Re: Python threading, and processes

Robin Haswell <rob@digital-crocus.com> wrote:
...[color=blue]
> Cheers for that info. The thread's main tasks are getting webpages
> (spidering), the actual amount of processing done in each thread is
> minimal - that's why I'm confused by the CPU usage.[/color]

BTW, spidering is a great use case for async (even-driven) programming;
use Twisted (or even build on top of asyncore) and you'll easily
saturate your network bandwidth with modest CPU demands.


Alex

[Yves Glodt]

Re: Python threading, and processes

Robin Haswell wrote:[color=blue]
> Hey there
>
> I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
> Debian machines, and I've noticed that if I open a lot of threads (say,
> 50), I get lots of python processes with individual PIDs, which consume a
> disproportionat e amount of CPU. Does this mean that Python is using the
> dummy_thread module by accident? And is there a realistic limitation to
> the number of threads I can do?[/color]

Though I can not answer your question, I have however a similar
situation here, on debian sarge.

I have a simple daemon that runs one thread, and I noticed that on our
sarges with kernel 2.4 my daemon creates 4 processes, on the ones with
kernel 2.6, only one process.

btw, I use the thread module.

best regards,
Yves
[color=blue]
> Cheers
>
> -Rob[/color]