Opposite of yield?

[achrist@easystreet.com]

Re: Opposite of yield?

Peter Hansen wrote:[color=blue]
>
> The real question might be why would you want to?
>
> If you don't want a thread, but you want something which takes
> input from elsewhere and does some processing, then returns
> control to some other place until more data is available (which
> is what one might assume if threads aren't good for you), there
> is already a convenient solution: the subroutine. ;-)
>
> Seriously, what's the requirement driving the need for this?
> (I suspect Erik Max's answer is still what you need, and you
> misunderstand the nature of what he was suggesting, but if you'll
> explain the specific case you have in mind we'll know for sure.)
>[/color]

No, I didn't say that I didn't want a thread. If there are no
serious drawbacks to relying heavily on threads, I'd be glad to
use them. This would be deluxe.

There really isn't any need for a generator, but it is a very
nice way to construct the input side of a program (the caller
doesn't have to worry about expicitly calling initialization and
finalization). Some kind of through-the-looking-glass version
of a generator would be a very nice way to construct the output
side.

I decided to test threads to see what happens:

-------------Warning, Code Follows---------------------

import Queue
import thread
import time

queues = [Queue.Queue()]
threads = []

appendLock = thread.allocate _lock()

def TransferData(qu eues, atPos):
while len(queues) < atPos + 1:
pass
inputQueue = queues[atPos]
outputQueue = Queue.Queue()
appendLock.acqu ire()
queues.append(o utputQueue)
appendLock.rele ase()
while 1:
data = inputQueue.get( True)
outputQueue.put (data, True)

startTime = time.time()
for i in range(1000):
print i,
threads.append( thread.start_ne w_thread(Transf erData, (queues, i)))
queues[0].put(i, True)
appendLock.acqu ire()
print queues[-1].get(True)
appendLock.rele ase()

print time.time() - startTime


--------------Warning Discussion Follows -------------------------

Running this on my Windows NT 200 NHz machine gave decent results.
With 1000 threads and 1000 Queues, memory used was 24MB. Python
takes almost 10 MB to start, so the overhead per thread + Queue
is only a little over 10 kb. Passing all the messages through
the chain took 2.5 minutes, so we are down around a quarter millisec
for each put or get.

That should be pretty good for many uses.


Al

[Jan Decaluwe]

Re: Opposite of yield?

achrist@easystr eet.com wrote:[color=blue]
>
> Peter Hansen wrote:[color=green]
> >
> >
> > Queue.Queue.get () ...
> >[/color]
>
> That looks good. But that needs a thread to block, right?
> A generator manages to run without being a thread (at least
> overtly). If we have the suck() statement, we also need
> something that's the opposite of a generator (a Sucker?)
> and something that's the opposite of an iterator (a Suckee?).
> I'm starting to get an idea why this is completely not all
> there.
>
> The main question this raises is "How lightweight are threads?"
> Can I program with dozens or hundreds of python threads in a
> program (for example under Windows) and not notice that this is
> an inefficient or inept coding style?
>
> Al[/color]

You may want to check out how such things are done with my MyHDL
package. The target application is hardware design, but the idea
is more probably general.

http://jandecaluwe.com/Tools/MyHDL/manual/model-obj.html

Jan

--
Jan Decaluwe - Resources bvba - http://jandecaluwe.com
Losbergenlaan 16, B-3010 Leuven, Belgium
Bored with EDA the way it is? Check this:

http://jandecaluwe.com/Tools/MyHDL/Overview.html

[Oren Tirosh]

Re: Opposite of yield?

On Wed, Sep 10, 2003 at 12:40:37PM -0700, achrist@easystr eet.com wrote:[color=blue]
> The yield statement looks to be a big step toward some kind of
> lightweight concurrency-oriented programming in python. Is there
> any similarly nice way to do the opposite of yield, just sit around
> (perhaps in the middle of a loop) and wait until some other routine
> (unknown to the waiting module) hurls a value in?[/color]

Here is a way of converting a generator function into a consumer using
a single item list as a mailbox:

from __future__ import generators

def consumer():
print "This code runs before any data is fed to the consumer"
mailbox = []; yield mailbox
while mailbox:
print "Here's what I got:", mailbox.pop()
yield mailbox
print "This code runs after end of data is signalled"

# How to feed a consumer:

i = 0
for mailbox in consumer():
if i < 5:
mailbox.append( i) # feed next item
i += 1


Output:
This code runs before any data is fed to the consumer
Here's what I got: 0
Here's what I got: 1
Here's what I got: 2
Here's what I got: 3
Here's what I got: 4
This code runs after end of data is signalled


For a more comprehensive solution try Christian Tismer's Stackless Python.

Oren

[Greg Chapman]

Re: Opposite of yield?

On Wed, 10 Sep 2003 12:40:37 -0700, achrist@easystr eet.com wrote:
[color=blue]
>
>The yield statement looks to be a big step toward some kind of
>lightweight concurrency-oriented programming in python. Is there
>any similarly nice way to do the opposite of yield, just sit around
>(perhaps in the middle of a loop) and wait until some other routine
>(unknown to the waiting module) hurls a value in?
>
>If this is not implemented now, is there any chance that it's on the
>list of things to come?
>[/color]

You can use a generator as the body of a function. That is, the generator is
only ever accessed through the function, and the function and the generator have
some agreed on way to pass parameters. I.e.:

def getline(output) :
'''getline accumulates text until it has a complete line (signaled
by either a newline (\\n) or a null (\\0)). It then passes the
line (without trailing \n) to output'''

genargs = [None]
def callgen(text):
genargs[0] = text
return gennext()
def gen():
line = ''
while True:
text, = genargs
if text:
lines = text.splitlines (True)
if line:
lines[0] = line+lines[0]
for line in lines[:-1]:
output(line[:-1])
line = lines[-1]
if line[-1] in '\n\0':
if line[-1] == '\n':
line = line[:-1]
output(line)
line = ''
yield None
gennext = gen().next
return callgen

def printer(text):
print text

lineprinter = getline(printer )

lineprinter('he llo') #nothing output
lineprinter(' world\n') #prints 'hello world'


---
Greg Chapman

[Hung Jung Lu]

Re: Opposite of yield?

achrist@easystr eet.com wrote in message news:<3F5F7E35. B5DA4669@easyst reet.com>...[color=blue]
> The yield statement looks to be a big step toward some kind of
> lightweight concurrency-oriented programming in python. Is there
> any similarly nice way to do the opposite of yield, just sit around
> (perhaps in the middle of a loop) and wait until some other routine
> (unknown to the waiting module) hurls a value in?[/color]

This sounds like event-driven programming to me.

(1) Usage of threads in principle is optional, but at Python level
event-driven programming is probably always done in multi-threaded
environment.

(2) Also, at software level, interruptions are always emulated by
polling (i.e. event loop.) You will have a loop somewhere.

(3) I won't get into details of the event-handler implementation, as
it is well-known and you probably already know it or can find it
easily. But I'd like to mention a simple implementation using
exception handling. For instance:

while 1:
try:
poll_keyboard()
poll_mouse()
poll_microphone ()
poll_temperatur e_sensor()
except IncomingSignal, e:
... handle event ...

You raise the IncomingSignal event inside the various polling
routines.

regards,

Hung Jung Lu

[Isaac To]

Re: Opposite of yield?

>>>>> "Hung" == Hung Jung Lu <hungjunglu@yah oo.com> writes:

Hung> This sounds like event-driven programming to me.

Hung> (1) Usage of threads in principle is optional, but at Python level
Hung> event-driven programming is probably always done in multi-threaded
Hung> environment.

That does not stop somebody who wants something cheaper than threads when he
know that at most one place of his program needs being serviced at each
time.

Hung> (2) Also, at software level, interruptions are always emulated by
Hung> polling (i.e. event loop.) You will have a loop somewhere.

At "software level", most event loops are waited by calling the OS. The OS
can deal with the hardware interrupts caused by devices or timers, or the
signalling caused by other processes. In these ways the OS will not need to
do polling most of the time.

Hung> (3) I won't get into details of the event-handler implementation,
Hung> as it is well-known and you probably already know it or can find
Hung> it easily. But I'd like to mention a simple implementation using
Hung> exception handling. For instance:

Hung> while 1: try: poll_keyboard() poll_mouse() poll_microphone ()
Hung> poll_temperatur e_sensor() except IncomingSignal, e: ... handle
Hung> event ...

Hung> You raise the IncomingSignal event inside the various polling
Hung> routines.

Never do that! You have a lot of ways to avoid holding up the CPU without
doing anything real. No event loop in real systems acts like that, except
for embedded systems that has exactly one program loaded into the memory at
the same time. Instead, use something like select.poll() to wait from more
than one source of input/output, or use a thread as suggested by the parent
thread.

Regards,
Isaac.

[Hung Jung Lu]

Re: Opposite of yield?

Isaac To <kkto@csis.hku. hk> wrote in message[color=blue]
> At "software level", most event loops are waited by calling the OS. The OS
> can deal with the hardware interrupts caused by devices or timers, or the
> signalling caused by other processes. In these ways the OS will not need to
> do polling most of the time.[/color]

OSes do polling most of the time: on their idle event loop. Yes,
"idle" is considered an event, too.

Also, events are not restricted to hardware inputs. Users can generate
their own events. Like in the original poster's example: once he
computes a result, he may want to generate an event, and let the
program handle the generated result through an event handler.
[color=blue]
> Never do that! You have a lot of ways to avoid holding up the CPU without
> doing anything real. No event loop in real systems acts like that, except
> for embedded systems that has exactly one program loaded into the memory at
> the same time. Instead, use something like select.poll() to wait from more
> than one source of input/output, or use a thread as suggested by the parent
> thread.[/color]

Take it easy. :) It's just a sample code that shows some fun usage of
exception handling in Python. What the original poster asked was
"opposite of yield", and I only wanted to point out that what he
wanted was probably just "event-driven programming", which I was
surprised that no one has mentioned, at all.

My example is just a typical dirty trick of execution flow control in
Python via the exception handling. It's a well-known trick.

I did not mention the details of typical event-driven programming
implementation, because it's known by most programmers. No need for me
to show it here.

regards,

Hung Jung

[Jacek Generowicz]

Re: Opposite of yield?

Bob Gailer <bgailer@alum.r pi.edu> writes:
[color=blue]
> Also reminiscent (a long time ago) someone proposed a COME FROM
> statement for FORTRAN to be the inverse of GO TO.[/color]

http://c2.com/cgi/wiki?ComeFrom