Parallel Python

[Nick Maclaren]

Re: Parallel Python


In article <1168597288.571 937.176430@s34g 2000cwa.googleg roups.com>,
"Paul Boddie" <paul@boddie.or g.ukwrites:
|parallelpyt... @gmail.com wrote:
|
| The main difference between MPI python solutions and ppsmp is that with
| MPI you have to organize both computations
| {MPI_Comm_rank( MPI_COMM_WORLD, &id); if id==1 then ... else ....} and
| data distribution (MPI_Send / MPI_Recv) by yourself. While with ppsmp
| you just submit a function with arguments to the execution server and
| retrieve the results later.
|>
|Couldn't you just provide similar conveniences on top of MPI? Searching
|for "Python MPI" yields a lot of existing work (as does "Python PVM"),
|so perhaps someone has already done so.

Yes. No problem.

|Also, what about various grid toolkits?

If you can find one that is robust enough for real work by someone who
is not deeply into developing Grid software, I will be amazed.


Regards,
Nick Maclaren.

[robert]

Re: Parallel Python

Paul Boddie wrote:

parallelpyt...@ gmail.com wrote:

>The main difference between MPI python solutions and ppsmp is that with
>MPI you have to organize both computations
>{MPI_Comm_rank (MPI_COMM_WORLD , &id); if id==1 then ... else ....} and
>data distribution (MPI_Send / MPI_Recv) by yourself. While with ppsmp
>you just submit a function with arguments to the execution server and
>retrieve the results later.

>
Couldn't you just provide similar conveniences on top of MPI? Searching
for "Python MPI" yields a lot of existing work (as does "Python PVM"),
so perhaps someone has already done so. Also, what about various grid
toolkits?
>
[...]
>

>Overall ppsmp is still work in progress and there are other interesting
>features which I would like to implement. This is the main reason why I
>do not open the source of ppsmp - to have better control of its future
>development, as advised here: http://en.wikipedia.org/wiki/Freeware :-)

>
Despite various probable reactions from people who will claim that
they're comfortable with binary-only products from a single vendor, I
think more people would be inclined to look at your software if you did
distribute the source code, even if they then disregarded what you've
done. My own experience with regard to releasing software is that even
with an open source licence, most people are likely to ignore your
projects than to suddenly jump on board and take control, and even if
your project somehow struck a chord and attracted a lot of interested
developers, would it really be such a bad thing? Many developers have
different experiences and insights which can only make your project
better, anyway.
>
Related to your work, I've released a parallel execution solution
called parallel/pprocess [1] under the LGPL and haven't really heard
about anyone really doing anything with it, let alone forking it and
showing my original efforts in a bad light. Perhaps most of the
downloaders believe me to be barking up the wrong tree (or just
barking) with the approach I've taken, but I think the best thing is to
abandon any fears of not doing things the best possible way and just be
open to improvements and suggestions.
>
Paul
>
[1] http://www.python.org/pypi/parallel

I'd be interested in an overview.
For ease of use a major criterion for me would be a pure python
solution, which also does the job of starting and controlling the
other process(es) automatically right (by default) on common
platforms.
Which of the existing (RPC) solutions are that nice?


Robert

[Neal Becker]

Re: Parallel Python

parallelpython@ gmail.com wrote:

Has anybody tried to run parallel python applications?
It appears that if your application is computation-bound using 'thread'
or 'threading' modules will not get you any speedup. That is because
python interpreter uses GIL(Global Interpreter Lock) for internal
bookkeeping. The later allows only one python byte-code instruction to
be executed at a time even if you have a multiprocessor computer.
To overcome this limitation, I've created ppsmp module:

http://www.parallelpython.com

It provides an easy way to run parallel python applications on smp
computers.
I would appreciate any comments/suggestions regarding it.
Thank you!
>

Looks interesting, but is there any way to use this for a cluster of
machines over a network (not smp)?

[Paul Boddie]

Re: Parallel Python

robert wrote:

Paul Boddie wrote:

[1] http://www.python.org/pypi/parallel

>
I'd be interested in an overview.

I think we've briefly discussed the above solution before, and I don't
think you're too enthusiastic about anything using interprocess
communication, which is what the above solution uses. Moreover, it's
intended as a threading replacement for SMP/multicore architectures
where one actually gets parallel execution (since it uses processes).

For ease of use a major criterion for me would be a pure python
solution, which also does the job of starting and controlling the
other process(es) automatically right (by default) on common
platforms.
Which of the existing (RPC) solutions are that nice?

Many people have nice things to say about Pyro, and there seem to be
various modules attempting parallel processing, or at least some kind
of job control, using that technology. See Konrad Hinsen's
ScientificPytho n solution for an example of this - I'm sure I've seen
others, too.

Paul

[Konrad Hinsen]

Re: Parallel Python

On Jan 12, 2007, at 11:21, Paul Boddie wrote:

done. My own experience with regard to releasing software is that even
with an open source licence, most people are likely to ignore your
projects than to suddenly jump on board and take control, and even if

My experience is exactly the same. And looking into the big world of
Open Source programs, the only case I ever heard of in which a
project was forked by someone else is the Emacs/XEmacs split. I'd be
happy if any of my projects ever reached that level of interest.

Related to your work, I've released a parallel execution solution
called parallel/pprocess [1] under the LGPL and haven't really heard
about anyone really doing anything with it, let alone forking it and

That's one more project... It seems that there is significant
interest in parallel computing in Python. Perhaps we should start a
special interest group? Not so much in order to work on a single
project; I believe that at the current state of parallel computing we
still need many different approaches to be tried. But an exchange of
experience could well be useful for all of us.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: hinsen@cnrs-orleans.fr
---------------------------------------------------------------------

[Paul Boddie]

Re: Parallel Python

Konrad Hinsen wrote:

>
That's one more project... It seems that there is significant
interest in parallel computing in Python. Perhaps we should start a
special interest group? Not so much in order to work on a single
project; I believe that at the current state of parallel computing we
still need many different approaches to be tried. But an exchange of
experience could well be useful for all of us.

I think a special interest group might be productive, but I've seen
varying levels of special interest in the different mailing lists
associated with such groups: the Web-SIG list started with enthusiasm,
produced a cascade of messages around WSGI, then dried up; the XML-SIG
list seems to be a sorry indication of how Python's XML scene has
drifted onto other matters; other such groups have also lost their
momentum.

It seems to me that a more useful first step would be to create an
overview of the different modules and put it on the python.org Wiki:

http://wiki.python.org/moin/FrontPage

http://wiki.python.org/moin/UsefulModules (a reasonable entry point)

If no-one beats me to it, I may write something up over the weekend.

Paul

[Konrad Hinsen]

Re: Parallel Python

On Jan 12, 2007, at 15:08, Paul Boddie wrote:

It seems to me that a more useful first step would be to create an
overview of the different modules and put it on the python.org Wiki:
>

http://wiki.python.org/moin/FrontPage

http://wiki.python.org/moin/UsefulModules (a reasonable entry point)
>
If no-one beats me to it, I may write something up over the weekend.

That sounds like a good idea. I won't beat you to it, but I'll have a
look next week and perhaps add information that I have.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: hinsen@cnrs-orleans.fr
---------------------------------------------------------------------

[mheslep@gmail.com]

Re: Parallel Python


Konrad Hinsen wrote:
..... Perhaps we should start a

special interest group? Not so much in order to work on a single
project; I believe that at the current state of parallel computing we
still need many different approaches to be tried. But an exchange of
experience could well be useful for all of us.
>

+ 1

-Mark

[parallelpython@gmail.com]

Re: Parallel Python

Looks interesting, but is there any way to use this for a cluster of

machines over a network (not smp)?

Networking capabilities will be included in the next release of
Parallel Python software (http://www.parallelpython.com), which is
coming soon.

Couldn't you just provide similar conveniences on top of MPI? Searching
for "Python MPI" yields a lot of existing work (as does "Python PVM"),
so perhaps someone has already done so.

Yes, it's possible to do it on the top of any environment which
supports IPC.

That's one more project... It seems that there is significant
interest in parallel computing in Python. Perhaps we should start a
special interest group? Not so much in order to work on a single
project; I believe that at the current state of parallel computing we
still need many different approaches to be tried. But an exchange of
experience could well be useful for all of us.

Well, I may just add that everybody is welcome to start discussion
regarding any parallel python project or idea in this forum:

http://www.parallelpython.com/component/option,com_smf/Itemid,29/board,2.0

[A.T.Hofkamp]

Distributed computation of jobs (was: Parallel Python)

On 2007-01-12, robert <no-spam@no-spam-no-spam.invalidwro te:

>>
>[1] http://www.python.org/pypi/parallel

>
I'd be interested in an overview.
For ease of use a major criterion for me would be a pure python
solution, which also does the job of starting and controlling the
other process(es) automatically right (by default) on common
platforms.

Let me add a few cents to the discussion with this announcement:

About three years ago, I wrote two Python modules, one called 'exec_proxy',
which uses ssh to run another exec_proxy instance at a remote machine, thus
providing ligh-weight transparent access to a machine across a network.

The idea behind this module was/is that by just using ssh you have network
transparency, much more light weight than most other distributed modules where
you have to start deamons at all machines.
Recently, the 'rthread' module was announced which takes the same approach (it
seems from the announcement). I have not compared both modules with each other.


The more interesting Python module called 'batchlib' lies on top of the former
(or any other module that provides transparency across the network). It
handles distribution of computation jobs in the form of a 'start-computation'
and 'get-results' pair of functions.

That is, you give it a set of machines it may use, you say to the entry-point,
compute for me this-and-this function with this-and-this parameters, and
batchlib does the rest.
(that is, it finds a free machine, copies the parameters over the network, runs
the job, the result is transported back, and you can get the result of a
computation by using the same (uniq) identification given by you when the job
was given to batchlib.)

We used it as computation backend for optimization problems, but since
'computation job' may mean anything, the module should be very generically
applicable.


Compared to most other parallel/distributed modules, I think that the other
modules more-or-less compare with exec_proxy (that is, they stop with
transparent network access), where exec_proxy was designed to have minimal
impact on required infra structure (ie just ssh or rsh which is generally
already available) and thus without many of the features available from the
other modules.

Batchlib starts where exec_proxy ends, namely lifting network primitives to the
level of providing a simple way of doing distributed computations (in the case
of exec_proxy, without adding network infra structure such as deamons).




Until now, both modules were used in-house, and it was not clear what we wanted
to do further with the software. Recently, we have decided that we have no
further use for this software (we think we want to move into a different
direction), clearing the way to release this software to the community.

You can get the software from my home page http://seweb.se.wtb.tue.nl/~hat
Both packages can be downloaded, and include documentation and an example.
The bad news is that I will not be able to do further development of these
modules. The code is 'end-of-life' for us.


Maybe you find the software useful,
Albert

[Paul Boddie]

Re: Distributed computation of jobs (was: Parallel Python)

A.T.Hofkamp skrev:

>
Let me add a few cents to the discussion with this announcement:

[Notes about exec_proxy, batchlib and rthread]

I've added entries for these modules, along with py.execnet, to the
parallel processing solutions page on the python.org Wiki:

http://wiki.python.org/moin/ParallelProcessing

Thanks for describing your work to us!

Paul

[robert]

Re: Distributed computation of jobs

Paul Boddie wrote:

A.T.Hofkamp skrev:

>Let me add a few cents to the discussion with this announcement:

>
[Notes about exec_proxy, batchlib and rthread]
>
I've added entries for these modules, along with py.execnet, to the
parallel processing solutions page on the python.org Wiki:
>

http://wiki.python.org/moin/ParallelProcessing

>
Thanks for describing your work to us!
>
Paul
>

as many libs are restriced to certain OS'es, and/or need/rely on
extension modules, few tags would probably improve: OS'es,
pure-python, dependeniess


Robert

[Paul Boddie]

Re: Distributed computation of jobs

robert wrote:

>
as many libs are restriced to certain OS'es, and/or need/rely on
extension modules, few tags would probably improve: OS'es,
pure-python, dependeniess

I've added some platform notes, although the library dependencies of
various MPI and PVM solutions are sort of obvious, but I'll get round
to adding those at some point unless you beat me to it. ;-)

Paul

[parallelpython@gmail.com]

Re: Parallel Python

On Jan 12, 11:52 am, Neal Becker <ndbeck...@gmai l.comwrote:

parallelpyt...@ gmail.com wrote:

Has anybody tried to runparallelpyth onapplications?
It appears that if your application is computation-bound using 'thread'
or 'threading' modules will not get you any speedup. That is because
pythoninterpret er uses GIL(Global Interpreter Lock) for internal
bookkeeping. The later allows only onepythonbyte-code instruction to
be executed at a time even if you have a multiprocessor computer.
To overcome this limitation, I've created ppsmp module:

http://www.parallelpython.com

It provides an easy way to runparallelpyth onapplications on smp
computers.
I would appreciate any comments/suggestions regarding it.
Thank you!

>
Looks interesting, but is there any way to use this for a cluster of
machines over a network (not smp)?

There are 2 major updates regarding Parallel Python: http://

https://www.parallelpython.com

1) Now (since version 1.2) parallel python software could be used for
cluster-wide parallelization (or even Internet-wide). It's also
renamed accordingly: pp (module is backward compatible with ppsmp)

2) Parallel Python became open source (under BSD license): http://

https://www.parallelpython.com/content/view/18/32/