Advanced concurrancy

[Peter Tillotson]

Hi,

I'm looking for an advanced concurrency module for python and don't seem
to be able to find anything suitable. Does anyone know where I might
find one? I know that there is CSP like functionality built into
Stackless but i'd like students to be able to use a standard python build.

I'm trying to develop distributed / Grid computing modules based on
python. The aim is to be able to use barriers for synchronisation and
channels for communication between processes running on a single box.
Then the jump to multiple processes on multiple boxes and eventually to
MPI implementations . Hopefully, each jump should not be that big a leap.

Of course it would be nice if there was a robust way of managing
concurrency in python aswell ;-)

p

[Calvin Spealman]

Re: Advanced concurrancy

Twisted [1] includes lots of support for asyncronous concurrency,
using deferreds. There is also the possiblity of PEP 342's [2]
concurrency through enhanced generators, and being able to pass data
to the generator every iteration. There are ways to simulate this, as
well. I've written a recipe [3] for it over at ASPN.

Personally, I like the generator approach better, because it gives a
feel of threads, but with direct control of the timeslicing, a better
sense of understanding, and doesn't rely on things happening in the
background, like deferreds often do.

On 7/28/05, Peter Tillotson <none@no.chance > wrote:[color=blue]
> Hi,
>
> I'm looking for an advanced concurrency module for python and don't seem
> to be able to find anything suitable. Does anyone know where I might
> find one? I know that there is CSP like functionality built into
> Stackless but i'd like students to be able to use a standard python build..
>
> I'm trying to develop distributed / Grid computing modules based on
> python. The aim is to be able to use barriers for synchronisation and
> channels for communication between processes running on a single box.
> Then the jump to multiple processes on multiple boxes and eventually to
> MPI implementations . Hopefully, each jump should not be that big a leap.
>
> Of course it would be nice if there was a robust way of managing
> concurrency in python aswell ;-)
>
> p
> --
> http://mail.python.org/mailman/listinfo/python-list
>[/color]

[jemfinch@gmail.com]

Re: Advanced concurrancy

Asynchrony is not concurrency. If you have to turn your code "inside
out," (that is, if you have to write your code such that the library
calls your code, rather than vice versa) it's very much *not*
concurrency: it's just asynchrony.

While Twisted makes asynchronous code relatively easy to write and
maintain, it's just not concurrency. I can't simply drop my
single-threaded code into it and have it work, like I can with a truly
concurrent system.

Jeremy

[Paolino]

Re: Advanced concurrancy

Peter Tillotson wrote:[color=blue]
> Hi,
>
> I'm looking for an advanced concurrency module for python and don't seem
> to be able to find anything suitable. Does anyone know where I might
> find one? I know that there is CSP like functionality built into
> Stackless but i'd like students to be able to use a standard python build.
>
> I'm trying to develop distributed / Grid computing modules based on
> python. The aim is to be able to use barriers for synchronisation and
> channels for communication between processes running on a single box.
> Then the jump to multiple processes on multiple boxes and eventually to
> MPI implementations . Hopefully, each jump should not be that big a leap.
>
> Of course it would be nice if there was a robust way of managing
> concurrency in python aswell ;-)
>[/color]


And deferredGenerat or in twisted.interne t.defer is the robust way for that.
It blows up python readability in contrast,but once you got them and
made your library I think they are also usable.
I do believe, without deferreds in the core ,python will have bad times
surviving the net, but that's really an opinion.

Have fun, Paolino

[Peter Tillotson]

Re: Advanced concurrancy

Cheers Guys,

I have come across twisted and used in async code. What i'm really
looking for is something that provides concurrency based on CSP or pi
calculus. Or something that looks much more like Java's JSR 166 which is
now integrated in Tiger.

Peter Tillotson wrote:[color=blue]
> Hi,
>
> I'm looking for an advanced concurrency module for python and don't seem
> to be able to find anything suitable. Does anyone know where I might
> find one? I know that there is CSP like functionality built into
> Stackless but i'd like students to be able to use a standard python build.
>
> I'm trying to develop distributed / Grid computing modules based on
> python. The aim is to be able to use barriers for synchronisation and
> channels for communication between processes running on a single box.
> Then the jump to multiple processes on multiple boxes and eventually to
> MPI implementations . Hopefully, each jump should not be that big a leap.
>
> Of course it would be nice if there was a robust way of managing
> concurrency in python aswell ;-)
>
> p[/color]

[Paul Rubin]

Re: Advanced concurrancy

Peter Tillotson <none@no.chance > writes:[color=blue]
> I have come across twisted and used in async code. What i'm really
> looking for is something that provides concurrency based on CSP or pi
> calculus. Or something that looks much more like Java's JSR 166 which
> is now integrated in Tiger.[/color]

Python doesn't have anything that fancy. You could look at:
pyro.sf.net - Python remote objects using sockets
poshmodule.sf.n et - objects shared between processes using shared memory
Queue module - synchronized queues for interthread communications
in one process

It would be nice if there were something like Queue that used MPI.
Underneath there could be either sockets, shared memory, some special
multiprocessor interconnect, or whatever.

[Michael Sparks]

Re: Advanced concurrancy

Peter Tillotson wrote:
[color=blue]
> Hi,
>
> I'm looking for an advanced concurrency module for python and don't seem
> to be able to find anything suitable. Does anyone know where I might
> find one? I know that there is CSP like functionality built into
> Stackless but i'd like students to be able to use a standard python build.[/color]

Please take a look at Kamaelia* - it /probably/ has what you're after by the
sounds of things. Currently the unit for sequential process can be either
generators or threads, and is single CPU, single process, however we do
expect to make the system multi-process and multi-system.
* http://kamaelia.sourceforge.net/

Currently it runs on Linux, Mac OS X, Windows and a subset works nicely on
Series 60 mobiles. (That has separate packaging) It works with standard
Python versions 2.2 and upwards.

The basic idea in Kamaelia is you have a class that represents a concurrent
unit that communicates with local interfaces only which are essentially
queues. The specific metaphor we use is that of an office worker with
inboxes and outboxes with deliveries made between outboxes to inboxes.
There also exists a simple environmental/service lookup facility which acts
like an assistant in the above metaphor, and has natural similarities to a
Linda type system.

(The actual rationale for the assistant facility though is based on
biological systems. We have communicating linked concurrent components -
which is much like many biological systems. However in addition to that
most biological systems also have a hormonal system - which is part of the
thinking behind the assistant system)

Generators (when embedded in a class) lend themselves very nicely to this
sort of model in our experience /because/ they are limited to a single
level (with regard to yield).

It's probably suitable for your students because we've tested the system on
pre-university trainees, and vacation trainees, and found they're able to
pick up the system, learn the basic ideas within a week or so (I have some
exercises on how to build a mini- version if that helps), and build
interesting systems.

For example we had a pre-university trainee start with us at the beginning
of the year, learn python, Kamaelia, and build a simple streaming system
taking a video file, taking snapshots and sending those to mobile phones
and PC's - this was over a period of 3 months. He'd only done a little bit
of access in the past, and a little bit of VB. Well that as well as a
simple learning system simulating a digital TV decode chain, but taking a
script instead of a transport stream.

We recently made a 0.2.0 release of the system (not announced on c.l.p yet)
that includes (basic) support for a wide range of multimedia/networked apps
which might help people getting started. Some interesting new additions in
the release are an IPython integration - allowing you to build Kamaelia
systems on the fly using a shell, much like you can build unix pipelines,
as well as a visual introspection tool (and network graph visualiser) which
allows you to see inside systems as they are running. (This has turned out
to be extremely useful - as you can expect with any CSP-type system)

The specific use cases you mention are also very closed aligned with our
aims for the project.

We're essentially working on making concurrency easy and natural to use,
(starting from the domain of networked multimedia). You can do incremental
development and transformation in exactly the way it sounds like you want,
and build interesting systems. We're also working on the assumption that if
you do that you can get performance later by specific optimisations (eg
more CPUs).
* Example of incremental component development here:
http://tinyurl.com/dp8n7

By the time we reach a 1.0 release (of Kamaelia) we're also aiming to be
able to integrate cleanly with Twisted (on Twisted's grounds), but it is
highly usable already - especially in your area. (In CVS we have tools for
building game type systems easily & Tk integration as well. Tk based CSP
systems are particularly fun to work with (as in fun, not "fun" :). One
project we are seriously looking at is a visual editor for these CSP-type
systems, since that appears now to be low hanging fruit.

We've got a white paper about Kamaelia here:
* http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml

This is "textualisation " of a presentation I gave at ACCU earlier in the
year and is an overview of the core areas of the system - hopefully enough
to let you know whether to look further!

I also gave an updated talk at Europython - the presentation for which can
be downloaded from here:
* http://www.python-in-business.org/ep...2589&track=692

Last week I also gave a more pragmatic, shorter talk at Open Tech which is
an introduction to Kamaelia, it's goals, and several examples of CSP type
systems ranging from simple audio clients/servers through to presentation
tools. That presentation can be downloaded from here:
* http://kamaelia.sourceforge.net/Kamaelia_OT2005.tgz

Both the Europython & Open tech bundles actually include the core
concurrency systems (called Axon), and the component modules (Kamaelia) in
the bundle, in addition to the slides and install docs. The presentation
tools included in the bundles are written in python using Kamaelia.

The project website is here: http://kamaelia.sourceforge.net/
[color=blue]
> I'm trying to develop distributed / Grid computing modules based on
> python. The aim is to be able to use barriers for synchronisation and
> channels for communication between processes running on a single box.
> Then the jump to multiple processes on multiple boxes and eventually to
> MPI implementations . Hopefully, each jump should not be that big a leap.
>
> Of course it would be nice if there was a robust way of managing
> concurrency in python aswell ;-)[/color]

It's getting there - the recent PEP regarding the ability to cause a
generator to have an exception thrown inside it provides very similar to
Unix signals as provided via "kill", which will be nice when 2.5 is
released.

Hopefully you find the system useful,

Best Regards,


Michael.
--
Michael Sparks, Senior R&D Engineer, Digital Media Group
Michael.Sparks@ rd.bbc.co.uk, http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This e-mail may contain personal views which are not the views of the BBC.

[Peter Tillotson]

Re: Advanced concurrancy

I've not yet had a chance to try some examples, but i've looked through
the documentation. It feels quite familiar, but i'd say that it is
closer to Jade, the fipa (federation of intelligent physical agents)
compliant agent framework than CSP or pi calculus. I like the behaviour
(component microthread) model, but the advantage of CSP / pi calculus is
that the resulting distributed system remains open to mathematical analysis.

For concurency its is the best framework i've seen :-) Have you come
across the pylinda tuplespace implementation. It might be well worth a
look. I might be barking up the wrong tree - but it seems to me that
there could be considerable overlap between tuplespaces and mailboxes,
though you did mention that you were moving towards twisted as the
underlying platform for the future.

I'm quite interested in the mini version and also using the modules as
mobile code rather than installing it formally. I'll probably zip the
Axion directory and distribute the zip with the code, adding the zip to
the python path dynamically.

cheers

p

Michael Sparks wrote:[color=blue]
> Peter Tillotson wrote:
>
>[color=green]
>>Hi,
>>
>>I'm looking for an advanced concurrency module for python and don't seem
>>to be able to find anything suitable. Does anyone know where I might
>>find one? I know that there is CSP like functionality built into
>>Stackless but i'd like students to be able to use a standard python build.[/color]
>
>
> Please take a look at Kamaelia* - it /probably/ has what you're after by the
> sounds of things. Currently the unit for sequential process can be either
> generators or threads, and is single CPU, single process, however we do
> expect to make the system multi-process and multi-system.
> * http://kamaelia.sourceforge.net/
>
> Currently it runs on Linux, Mac OS X, Windows and a subset works nicely on
> Series 60 mobiles. (That has separate packaging) It works with standard
> Python versions 2.2 and upwards.
>
> The basic idea in Kamaelia is you have a class that represents a concurrent
> unit that communicates with local interfaces only which are essentially
> queues. The specific metaphor we use is that of an office worker with
> inboxes and outboxes with deliveries made between outboxes to inboxes.
> There also exists a simple environmental/service lookup facility which acts
> like an assistant in the above metaphor, and has natural similarities to a
> Linda type system.
>
> (The actual rationale for the assistant facility though is based on
> biological systems. We have communicating linked concurrent components -
> which is much like many biological systems. However in addition to that
> most biological systems also have a hormonal system - which is part of the
> thinking behind the assistant system)
>
> Generators (when embedded in a class) lend themselves very nicely to this
> sort of model in our experience /because/ they are limited to a single
> level (with regard to yield).
>
> It's probably suitable for your students because we've tested the system on
> pre-university trainees, and vacation trainees, and found they're able to
> pick up the system, learn the basic ideas within a week or so (I have some
> exercises on how to build a mini- version if that helps), and build
> interesting systems.
>
> For example we had a pre-university trainee start with us at the beginning
> of the year, learn python, Kamaelia, and build a simple streaming system
> taking a video file, taking snapshots and sending those to mobile phones
> and PC's - this was over a period of 3 months. He'd only done a little bit
> of access in the past, and a little bit of VB. Well that as well as a
> simple learning system simulating a digital TV decode chain, but taking a
> script instead of a transport stream.
>
> We recently made a 0.2.0 release of the system (not announced on c.l.p yet)
> that includes (basic) support for a wide range of multimedia/networked apps
> which might help people getting started. Some interesting new additions in
> the release are an IPython integration - allowing you to build Kamaelia
> systems on the fly using a shell, much like you can build unix pipelines,
> as well as a visual introspection tool (and network graph visualiser) which
> allows you to see inside systems as they are running. (This has turned out
> to be extremely useful - as you can expect with any CSP-type system)
>
> The specific use cases you mention are also very closed aligned with our
> aims for the project.
>
> We're essentially working on making concurrency easy and natural to use,
> (starting from the domain of networked multimedia). You can do incremental
> development and transformation in exactly the way it sounds like you want,
> and build interesting systems. We're also working on the assumption that if
> you do that you can get performance later by specific optimisations (eg
> more CPUs).
> * Example of incremental component development here:
> http://tinyurl.com/dp8n7
>
> By the time we reach a 1.0 release (of Kamaelia) we're also aiming to be
> able to integrate cleanly with Twisted (on Twisted's grounds), but it is
> highly usable already - especially in your area. (In CVS we have tools for
> building game type systems easily & Tk integration as well. Tk based CSP
> systems are particularly fun to work with (as in fun, not "fun" :). One
> project we are seriously looking at is a visual editor for these CSP-type
> systems, since that appears now to be low hanging fruit.
>
> We've got a white paper about Kamaelia here:
> * http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml
>
> This is "textualisation " of a presentation I gave at ACCU earlier in the
> year and is an overview of the core areas of the system - hopefully enough
> to let you know whether to look further!
>
> I also gave an updated talk at Europython - the presentation for which can
> be downloaded from here:
> * http://www.python-in-business.org/ep...2589&track=692
>
> Last week I also gave a more pragmatic, shorter talk at Open Tech which is
> an introduction to Kamaelia, it's goals, and several examples of CSP type
> systems ranging from simple audio clients/servers through to presentation
> tools. That presentation can be downloaded from here:
> * http://kamaelia.sourceforge.net/Kamaelia_OT2005.tgz
>
> Both the Europython & Open tech bundles actually include the core
> concurrency systems (called Axon), and the component modules (Kamaelia) in
> the bundle, in addition to the slides and install docs. The presentation
> tools included in the bundles are written in python using Kamaelia.
>
> The project website is here: http://kamaelia.sourceforge.net/
>
>[color=green]
>>I'm trying to develop distributed / Grid computing modules based on
>>python. The aim is to be able to use barriers for synchronisation and
>>channels for communication between processes running on a single box.
>>Then the jump to multiple processes on multiple boxes and eventually to
>>MPI implementations . Hopefully, each jump should not be that big a leap.
>>
>>Of course it would be nice if there was a robust way of managing
>>concurrency in python aswell ;-)[/color]
>
>
> It's getting there - the recent PEP regarding the ability to cause a
> generator to have an exception thrown inside it provides very similar to
> Unix signals as provided via "kill", which will be nice when 2.5 is
> released.
>
> Hopefully you find the system useful,
>
> Best Regards,
>
>
> Michael.
> --
> Michael Sparks, Senior R&D Engineer, Digital Media Group
> Michael.Sparks@ rd.bbc.co.uk, http://kamaelia.sourceforge.net/
> British Broadcasting Corporation, Research and Development
> Kingswood Warren, Surrey KT20 6NP
>
> This e-mail may contain personal views which are not the views of the BBC.
>[/color]

[Calvin Spealman]

Re: Advanced concurrancy

On 28 Jul 2005 10:41:54 -0700, jemfinch@gmail. com <jemfinch@gmail .com> wrote:[color=blue]
> Asynchrony is not concurrency. If you have to turn your code "inside
> out," (that is, if you have to write your code such that the library
> calls your code, rather than vice versa) it's very much *not*
> concurrency: it's just asynchrony.
>
> While Twisted makes asynchronous code relatively easy to write and
> maintain, it's just not concurrency. I can't simply drop my
> single-threaded code into it and have it work, like I can with a truly
> concurrent system.
>
> Jeremy[/color]

When you can ever just "simply drop" any single-threaded code into an
enviroment where it is sharing the resources and data with other
executing code simulataniously , it just "have it work", that will be
the day. Unfortunately, in practice, this simply is not how things
work. For code to operate peacefully together, it must be designed to
do so. Even when code is running in seperate processes, they must work
together to share some resources, and that is simply the way of
things. Concurrency can not (and perhaps should not) be an automatic
fix-all pill.

[Michael Sparks]

Re: Advanced concurrancy

Peter Tillotson wrote:
[color=blue]
> I've not yet had a chance to try some examples, but i've looked through
> the documentation. It feels quite familiar,[/color]

It hopefully should. The approach is based essentially on an asynchronous
hardware approach, on the recognition that the fundamental reason that
hangs together is because it can be reasoned about in a similar manner to
CSP. (That naturally leads to similarities of course with Occam)
[color=blue]
> but i'd say that it is closer to Jade, the fipa (federation of intelligent
> physical agents) compliant agent framework[/color]

Thanks - I've not come across those - I'll have to take a look at that in
detail. (With a cursory look, they sounds similar to an application area a
colleague is interested in investigating)
[color=blue]
> than CSP or pi calculus. I like the behaviour
> (component microthread) model, but the advantage of CSP / pi calculus is
> that the resulting distributed system remains open to mathematical
> analysis.[/color]

Indeed. However, by trying to have the same basic limitations in the system,
Kamaelia should be amenable to the sort of approaches taken by hardware
verification systems. It's possibly worth mentioning that there is one
person at BBC R&D who is extremely interested in applying CSP style
analysis techniques to Kamaelia systems whom I sure would be interested in
chatting to you. (Indeed the similarities between Kamaelia and CSP is
biggest reason for his interest :-)

The idea of a verification system for software that is actually in use for
real world systems would be extremely nice/useful to have (though not
simple to achieve).

FWIW, I am interested in hearing suggestions on changing the system to make
it more amenable to mathematical analysis. (Largely because I'm a fan of
formal methods where possible, but pragmatically view TDD as the current
most practical way of getting towards the levels of reliabiliy aimed for by
formal methods)
[color=blue]
> For concurency its is the best framework i've seen :-)[/color]

Thanks!
[color=blue]
> Have you come across the pylinda tuplespace implementation. It might be
> well worth a look.[/color]

I have, and I liked what I saw :)
[color=blue]
> I might be barking up the wrong tree - but it seems to me that
> there could be considerable overlap between tuplespaces and mailboxes,[/color]

Probably the closer match is the co-ordinating assistant tracker (I can
see what you mean though). That provides a basic global key/value
store/retrieve service, which in a concurrent system is probably most
akin to a Linda tuplespace. In a biological system it's more akin to the
hormonal system which is more of the model I like to think of as to /when/
to use that part of the system. (Evolution having more 'experience' than
me on how to build a massively concurrent system after all !!)

It's generally only used at present for one of two purposes:
* Finding services that already exist (so that we don't try to open two
displays for example)
* Stats collection.

Using a Linda tuplespace there would be an interesting next move, however
we generally only extend the system based on specific need rather than
exploring architecture (Which I'm sure you understand!).
[color=blue]
> though you did mention that you were moving towards twisted as the
> underlying platform for the future.[/color]

Perhaps not underlying, but definitely interoperating with (It seems rather
wasteful to reinvent wheels).
[color=blue]
> I'm quite interested in the mini version and also using the modules as
> mobile code rather than installing it formally.[/color]

I'll document it slightly better and post up on the website in the next 48
hours or so. In the meantime, the optimisation test which includes a
slightly modified mini-axon can be found here:

http://cvs.sourceforge.net/viewcvs.py/kamaelia/Sketches/OptimisationTest/

Specifically the file "simplegame.py" . This includes a cutdown version of
Axon that's about 100 lines long (lines 41 - 144 specifically), with the
important classes being:

class microprocess(ob ject):
class newComponent(ob ject):
class scheduler(micro process):
class component(micro process):
class send_one_compon ent(component):
def linkage(source, sink):
[color=blue]
> I'll probably zip the Axon directory and distribute the zip with the code,
> adding the zip to the python path dynamically.[/color]

Sounds interesting. If you go ahead with this we'd be interested in hearing
how you get on.

Best Regards,


Michael.

[Stefan Rank]

Re: Advanced concurrancy

on 04.08.2005 00:36 Michael Sparks said the following:[color=blue]
> Peter Tillotson wrote:[color=green]
>>I'm quite interested in the mini version and also using the modules as
>>mobile code rather than installing it formally.[/color]
>
> I'll document it slightly better and post up on the website in the next 48
> hours or so. In the meantime, the optimisation test which includes a
> slightly modified mini-axon can be found here:
>
> http://cvs.sourceforge.net/viewcvs.p...imisationTest/
>
> Specifically the file "simplegame.py" .[/color]

Hi, sorry to intrude here.
I looked at Kamaelia (Axon actually) recently and I liked it,
what I did miss in the code (maybe I just did not find it) is a blocking
way for send-ing and recv-ing from in/outboxes that the scheduler is
aware of.
It seems as though all components basically have to do busy waiting now.

So do you plan on including a kind of scheduler-aware blocking
communication (like the `channels` of the `tasklets` in stackless)?

(as you have a postman already - not a bad idea i think if you compare
with multi-agent-systems theory (see jade) - it could be responsible for
alerting the scheduler of box.has_data)

Cheers,
Stefan

[Matt Hammond]

Re: Advanced concurrancy

Hi Stefan,
[color=blue]
> It seems as though all components basically have to do busy waiting now.[/color]

You are right - components are, for the most part, busy-waiting. Which
is not a good thing!
[color=blue]
> So do you plan on including a kind of scheduler-aware blocking
> communication (like the `channels` of the `tasklets` in stackless)?
>
> (as you have a postman already - not a bad idea i think if you compare
> with multi-agent-systems theory (see jade) - it could be responsible for
> alerting the scheduler of box.has_data)[/color]

There is basic support for components to block, by calling the
self.pause() method. From the next yield, this blocks further execution
until
data arrives on any inbox.

The un-pausing action is actually performed by code within the
component
itself, though this is in effect, as you suggested it might be,
instigated by
the postman microprocess.

A simple example would be an improved 'consoleEcho' component:

class consoleEchoer(A xon.Component.c omponent):
....
def main(self):
while 1:
yield 1
while self.dataReady( "inbox"):
print self.recv("inbo x")
self.pause()

That said, if you look at the code, you'll see that probably the
majority of components do not make good use of this yet!

There are quite probably better mechanisms around - hence the
distinction, in the code, between microprocesses and components.
We'd be very interested if yourself or other people want to play with
alternative communication & blocking mechanisms.

I'd hope that even if the components and postboxes model doesn't
work out in the long run, the underlying generator based
microprocesses code could still be of some value to others!


regards


Matt

[Matt Hammond]

Re: Advanced concurrancy

Ooops! In case you're wondering who I am, I'm working
on Kamaelia with Michael Sparks at the BBC.


Matt

[Stefan Rank]

Re: Advanced concurrancy

on 04.08.2005 11:15 Matt Hammond said the following:[color=blue]
> Hi Stefan,
>[color=green]
>>It seems as though all components basically have to do busy waiting now.[/color]
>
> You are right - components are, for the most part, busy-waiting. Which
> is not a good thing!
>[color=green]
>>So do you plan on including a kind of scheduler-aware blocking
>>communicati on (like the `channels` of the `tasklets` in stackless)?
>>[/color][/color]
[snip][color=blue]
>
> There is basic support for components to block, by calling the
> self.pause() method. From the next yield, this blocks further execution
> until
> data arrives on any inbox.
>[/color]
[snip]

ah, i was searching for blocking on a specific inbox, but this is
actually a saner approach i think. why should you block on one inbox and
ignore data coming in on another...
[color=blue]
> There are quite probably better mechanisms around - hence the
> distinction, in the code, between microprocesses and components.
> We'd be very interested if yourself or other people want to play with
> alternative communication & blocking mechanisms.[/color]

i think in-out channels is probably the best way for easy to manage
comunicating processes.
[color=blue]
> I'd hope that even if the components and postboxes model doesn't
> work out in the long run, the underlying generator based
> microprocesses code could still be of some value to others![/color]

yes both levels will (it will take some time before i can use it
here...), i think of it as a more complete version of some of the
stackless features in standard python. (thanks again by the way)
the magic.greenlets have 'cooperative microprocesses' , but lack the more
pipe-like communication facilities and the independency of the
generators+sche duler based approach.
the coroutine support in future pythons (see the generator PEPs) will
probably make this easier... maybe::

next_inbox_to_r ead = yield blockon(inbox1, inbox2)

this communication infrastructure is definitely a Good Thing (TM), the
same holds for a standard environment or registry (wassitsname, your CAT
thing) as you need at least one well-known point of reference in a
'distributed' system.

<dreamingmode >
now if someone could only cut the gordian knot of the
GlobalInterpret erLock to make all this really concurrent... and then,
when she's at it, allow transparent distribution of components in a
network... ;-)
</dreamingmode>

[Matt Hammond]

Re: Advanced concurrancy

Hi Stefan,

On Fri, 05 Aug 2005 07:20:36 +0100, Stefan Rank <stefan.rank@of ai.at>
wrote:
[color=blue][color=green][color=darkred]
>>> So do you plan on including a kind of scheduler-aware blocking
>>> communication (like the `channels` of the `tasklets` in stackless)?[/color][/color][/color]

Not at the moment. The kinds of applications we've mainly been looking at,
whilst developing, have been multimedia data-flow oriented, so blocking
communication would be used as a rate limiting mechanism. However, we've
found we can achieve that by putting explicit feedback links in.

I think we are also trying to stick with the asynchronous way, if
possible, because it may well make things simpler and more efficient when
it comes to distributing components across processes / networked
computers. Synchronous links will lead to handshaking latencies, more data
needing to flow etc. IMO that kind of facility could always be built on
top - for example - using a feedback inbox->outbox path to 'request' new
data ... if you really need it.

.... it also simplifies the scheduler. We've tried, so far, to keep the
underlying microprocess model decoupled from the components and
inbox/outbox layer. That said, that may have to change eventually!
[color=blue]
> this communication infrastructure is definitely a Good Thing (TM), the
> same holds for a standard environment or registry (wassitsname, your CAT
> thing) as you need at least one well-known point of reference in a
> 'distributed' system.[/color]

yep its a CAT (Coordinating Assistant Tracker). Nothing to do with the
project logo ... nooo! :-)
[color=blue]
> <dreamingmode >
> now if someone could only cut the gordian knot of the
> GlobalInterpret erLock to make all this really concurrent... and then,
> when she's at it, allow transparent distribution of components in a
> network... ;-)
> </dreamingmode>[/color]

I'll second that!

regards


Matt
--

| Matt Hammond
| R&D Engineer, BBC Research and Development, Tadworth, Surrey, UK.