threading support in python

[km]

Hi all,

Is there any PEP to introduce true threading features into python's
next version as in java? i mean without having GIL.
when compared to other languages, python is fun to code but i feel its
is lacking behind in threading

regards,
KM

[bayerj]

Re: threading support in python

Hi,

GIL won't go. You might want to read
http://blog.ianbicking.org/gil-of-doom.html .

Regards,
-Justin

[km]

Re: threading support in python

Hi all,
Are there any alternate ways of attaining true threading in python ?
if GIL doesnt go then does it mean that python is useless for
computation intensive scientific applications which are in need of
parallelization in threading context ?

regards,
KM
---------------------------------------------------------------------------
On 4 Sep 2006 07:58:00 -0700, bayerj <bayerj@in.tum. dewrote:

Hi,
>
GIL won't go. You might want to read
http://blog.ianbicking.org/gil-of-doom.html .
>
Regards,
-Justin
>
--

http://mail.python.org/mailman/listinfo/python-list

>

[bayerj]

Re: threading support in python

Hi,

You might want to split your calculation onto different
worker-processes.

Then you can use POSH [1] to share data and objects.
You might even want to go a step further and share the data via
Sockets/XML-RPC or something like that. That makes it easy to throw
aditional boxes at a specific calculation, because it can be set up in
about no time.
You can even use Twisted Spread [2] and its perspective broker to do
this on a higher level.

If that's not what you want, you are left with Java I guess.

Regards,
-Justin

[1] http://poshmodule.sourceforge.net/
[2] http://twistedmatrix.com/projects/co.../howto/pb.html

[Richard Brodie]

Re: threading support in python


"km" <srikrishnamoha n@gmail.comwrot e in message
news:mailman.10 337.1157383940. 27775.python-list@python.org ...

if GIL doesnt go then does it mean that python is useless for
computation intensive scientific applications which are in need of
parallelization in threading context ?

No.

[Sybren Stuvel]

Re: threading support in python

km enlightened us with:

Is there any PEP to introduce true threading features into python's
next version as in java? i mean without having GIL.

What is GIL? Except for the Dutch word for SCREAM that is...

when compared to other languages, python is fun to code but i feel
its is lacking behind in threading

What's wrong with the current threading? AFAIK it's directly linked to
the threading of the underlying platform.

Sybren
--
Sybren Stüvel
Stüvel IT - http://www.stuvel.eu/

[Diez B. Roggisch]

Re: threading support in python

Sybren Stuvel wrote:

km enlightened us with:

>Is there any PEP to introduce true threading features into python's
>next version as in java? i mean without having GIL.

>
What is GIL? Except for the Dutch word for SCREAM that is...

the global interpreter lock, that prevents python from concurrently
modifying internal structures causing segfaults.

>when compared to other languages, python is fun to code but i feel
>its is lacking behind in threading

>
What's wrong with the current threading? AFAIK it's directly linked to
the threading of the underlying platform.

There exist rare cases (see the link from bayerj) where the GIL is an
annoyance, and with the dawn of MP-cores all over the place it might be
considered a good idea removing it - maybe. But I doubt that is something
to be considered for py2.x

Diez

[Sandra-24]

Re: threading support in python

The trouble is there are some environments where you are forced to use
threads. Apache and mod_python are an example. You can't make use of
mutliple CPUs unless you're on *nux and run with multiple processes AND
you're application doesn't store large amounts of data in memory (which
mine does) so you'd have to physically double the computer's memory for
a daul-core, or quadruple it for a quadcore. And forget about running a
windows server, apache will not even run with multiple processes.

In years to come this will be more of an issue because single core CPUs
will be harder to come by, you'll be throwing away half of every CPU
you buy.

-Sandra

[Daniel Dittmar]

Re: threading support in python

km wrote:

Is there any PEP to introduce true threading features into python's
next version as in java? i mean without having GIL.
when compared to other languages, python is fun to code but i feel its
is lacking behind in threading

Some of the technical problems:

- probably breaks compatibility of extensions at the source level in a
big way, although this might be handled by SWIG, boost and other code
generators
- reference counting will have to be synchronized, which means that
Python will become slower
- removing reference counting and relying on garbage collection alone
will break many Python applications (because they rely on files being
closed at end of scope etc.)

Daniel

[Rob Williscroft]

Re: threading support in python

Daniel Dittmar wrote in news:edhl07$b2t $1@news.sap-ag.de in
comp.lang.pytho n:

- removing reference counting and relying on garbage collection alone
will break many Python applications (because they rely on files being
closed at end of scope etc.)
>

They are already broken on at least 2 python implementations , so
why worry about another one.

Rob.
--

http://www.victim-prime.dsl.pipex.com/

[sjdevnull@yahoo.com]

Re: threading support in python

Sandra-24 wrote:

The trouble is there are some environments where you are forced to use
threads. Apache and mod_python are an example. You can't make use of
mutliple CPUs unless you're on *nux and run with multiple processes AND
you're application doesn't store large amounts of data in memory (which
mine does) so you'd have to physically double the computer's memory for
a daul-core, or quadruple it for a quadcore.

You seem to be confused about the nature of multiple-process
programming.

If you're on a modern Unix/Linux platform and you have static read-only
data, you can just read it in before forking and it'll be shared
between the processes..

If it's read/write data or you're not on a Unix platform, you can use
shared memory to shared it between many processes.

Threads are way overused in modern multiexecution programming. The
decision on whether to use processes or threads should come down to
whether you want to share everything, or whether you have specific
pieces of data you want to share. With processes + shm, you can gain
the security of protected memory for the majority of your code + data,
only sacrificing it where you need to share the data.

The entire Windows programming world tends to be so biased toward
multithreading that they often don't even acknowledge the existence of
generally superior alternatives. I think that's in large part because
historically on Windows 3.1/95/98 there was no good way to create
processes without running a new binary, and so a culture of threading
grew up. Even today many Windows programmers are unfamiliar with using
CreateProcessEx with SectionHandle=N ULL for efficient copy-on-write
process creation.

And forget about running a
windows server, apache will not even run with multiple processes.

It used to run on windows with multiple processes. If it really won't
now, use an older version or contribute a fix.

Now, the GIL is independent of this; if you really need threading in
your situation (you share almost everything and have hugely complex
data structures that are difficult to maintain in shm) then you're
still going to run into GIL serialization. If you're doing a lot of
work in native code extensions this may not actually be a big
performance hit, if not it can be pretty bad.

[Paul Rubin]

Re: threading support in python

"sjdevnull@yaho o.com" <sjdevnull@yaho o.comwrites:

If it's read/write data or you're not on a Unix platform, you can use
shared memory to shared it between many processes.
>
Threads are way overused in modern multiexecution programming. The
decision on whether to use processes or threads should come down to
whether you want to share everything, or whether you have specific
pieces of data you want to share.

Shared memory means there's a byte vector (the shared memory region)
accessible to multiple processes. The processes don't use the same
machine addresses to reference the vector. Any data structures
(e.g. those containing pointers) shared between the processes have to
be marshalled in and out of the byte vector instead of being accessed
normally. Any live objects such as open sockets have to be shared
some other way. It's not a matter of sharing "everything "; shared
memory is a pain in the neck even to share a single object. These
things really can be easier with threads.

[Daniel Dittmar]

Re: threading support in python

Rob Williscroft wrote:

Daniel Dittmar wrote in news:edhl07$b2t $1@news.sap-ag.de in
comp.lang.pytho n:
>
>

>>- removing reference counting and relying on garbage collection alone
>>will break many Python applications (because they rely on files being
>>closed at end of scope etc.)
>>

>
>
They are already broken on at least 2 python implementations , so
why worry about another one.

I guess few applications or libraries are being ported from CPython to
Jython or IronPython as each is targeting a different standard library,
so this isn't that much of a problem yet.

Daniel

[Sandra-24]

Re: threading support in python

You seem to be confused about the nature of multiple-process

programming.
>
If you're on a modern Unix/Linux platform and you have static read-only
data, you can just read it in before forking and it'll be shared
between the processes..

Not familiar with *nix programming, but I'll take your word on it.

If it's read/write data or you're not on a Unix platform, you can use
shared memory to shared it between many processes.

I know how shared memory works, it's the last resort in my opinion.

Threads are way overused in modern multiexecution programming. The

<snip>

It used to run on windows with multiple processes. If it really won't
now, use an older version or contribute a fix.

First of all I'm not in control of spawning processes or threads.
Apache does that, and apache has no MPM for windows that uses more than
1 process. Secondly "Superior" is definately a matter of opinion. Let's
see how you would define superior.

1) Port (a nicer word for rewrite) the worker MPM from *nix to Windows.
2) Alternately switch to running Linux servers (which have their
plusses) but about which I know nothing. I've been using Windows since
I was 10 years old, I'm confident in my ability to build, secure, and
maintain a Windows server. I don't think anyone would recommend me to
run Linux servers with very little in the way of Linux experience.
3) Rewrite my codebase to use some form of shared memory. This would be
a terrible nightmare that would take at least a month of development
time and a lot of heavy rewriting. It would be very difficult, but I'll
grant that it may work if done properly with only small performance
losses. Sounds like a deal.

I would find an easier time, I think, porting mod_python to .net and
leaving that GIL behind forever. Thankfully, I'm not considering such
drastic measures - yet.

Why on earth would I want to do all of that work? Just because you want
to keep this evil thing called a GIL? My suggestion is in python 3
ditch the ref counting, use a real garbage collector, and make that GIL
walk the plank. I have my doubts that it would happen, but that's fine,
the future of python is in things like IronPython and PyPy. CPython's
days are numbered. If there was a mod_dotnet I wouldn't be using
CPython anymore.

Now, the GIL is independent of this; if you really need threading in
your situation (you share almost everything and have hugely complex
data structures that are difficult to maintain in shm) then you're
still going to run into GIL serialization. If you're doing a lot of
work in native code extensions this may not actually be a big
performance hit, if not it can be pretty bad.

Actually, I'm not sure I understand you correctly. You're saying that
in an environment like apache (with 250 threads or so) and my hugely
complex shared data structures, that the GIL is going to cause a huge
performance hit? So even if I do manage to find my way around in the
Linux world, and I upgrade my memory, I'm still going to be paying for
that darned GIL?

Will the madness never end?
-Sandra

[Steve Holden]

Re: threading support in python

Sandra-24 wrote:
[Sandra understands shared memory]

>
I would find an easier time, I think, porting mod_python to .net and
leaving that GIL behind forever. Thankfully, I'm not considering such
drastic measures - yet.
>

Quite right too. You haven't even sacrificed a chicken yet ...

Why on earth would I want to do all of that work? Just because you want
to keep this evil thing called a GIL? My suggestion is in python 3
ditch the ref counting, use a real garbage collector, and make that GIL
walk the plank. I have my doubts that it would happen, but that's fine,
the future of python is in things like IronPython and PyPy. CPython's
days are numbered. If there was a mod_dotnet I wouldn't be using
CPython anymore.
>

You write as though the GIL was invented to get in the programmer's way,
which is quite wrong. It's there to avoid deep problems with thread
interaction. Languages that haven't bitten that bullet can bite you in
quite nasty ways when you write threaded applications.

Contrary to your apparent opinion, the GIL has nothing to do with
reference-counting.

>

>>Now, the GIL is independent of this; if you really need threading in
>>your situation (you share almost everything and have hugely complex
>>data structures that are difficult to maintain in shm) then you're
>>still going to run into GIL serialization. If you're doing a lot of
>>work in native code extensions this may not actually be a big
>>performance hit, if not it can be pretty bad.

>
>
Actually, I'm not sure I understand you correctly. You're saying that
in an environment like apache (with 250 threads or so) and my hugely
complex shared data structures, that the GIL is going to cause a huge
performance hit? So even if I do manage to find my way around in the
Linux world, and I upgrade my memory, I'm still going to be paying for
that darned GIL?
>

I think the suggestion was rather that abandoning Python because of the
GIL might be premature optimisation. But since you appear to be sticking
with it, that might have been unnecessary advice.

Will the madness never end?

This reveals an opinion of the development team that's altogether too
low. I believe the GIL was introduced for good reasons.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden