creating generators from function

[Simon Wittber]

I use a coroutine/generator framework for simulating concurrent processes.

To do this, I write all my functions using the general form:

while True:
do stuff
yield None

To make these generator functions compatible with a standard thread
interface, I attempted to write a decorator which converts a standard
function into a generator function (code attached below).
Unfortunately, I received an exception, before I could test if my idea
would work:

TypeError: cannot create 'generator' instances

I guess I can go the other way, and convert a generator into a
function, but that just doesn't seem right.

Is there anyway solve the problem described?

Sw.


from opcode import opmap
import types
globals().updat e(opmap)

def generatorize(f) :
co = f.func_code
nc = [ord(x) for x in co.co_code]
for op,i in enumerate(nc):
if op == RETURN_VALUE:
nc[i] = YIELD_VALUE
newcode = ''.join([chr(x) for x in nc])
codeobj = type(co)(co.co_ argcount, co.co_nlocals, co.co_stacksize ,
co.co_flags, newcode, co.co_consts, co.co_names,
co.co_varnames, co.co_filename, co.co_name,
co.co_firstline no, co.co_lnotab, co.co_freevars,
co.co_cellvars)

print types.Generator Type(f)(codeobj , f.func_globals, f.func_name,
f.func_defaults , f.func_closure)
return f

@generatorize
def f():
while True:
return 1

def g():
while True:
yield 1

print g()
print f()

[Steven Bethard]

Re: creating generators from function

Simon Wittber wrote:[color=blue]
> I use a coroutine/generator framework for simulating concurrent processes.
>
> To do this, I write all my functions using the general form:
>
> while True:
> do stuff
> yield None
>
> To make these generator functions compatible with a standard thread
> interface, I attempted to write a decorator which converts a standard
> function into a generator function (code attached below).[/color]
[snip][color=blue]
> @generatorize
> def f():
> while True:
> return 1
>
> def g():
> while True:
> yield 1
>
> print g()
> print f()[/color]

I'm a little confused as to what you're trying to do here. The f()
function, in particular, doesn't make much sense -- the 'while True'
doesn't do anything since the 'return 1' is executed on the first time
through the loop. If you really do want to turn your functions into
generators of the form:

while True:
do stuff
yield None

why can't you just do:
[color=blue][color=green][color=darkred]
>>> def generatorize(f) :[/color][/color][/color]
.... def new_f(*args, **kwds):
.... while True:
.... f(*args, **kwds)
.... yield None
.... return new_f
....[color=blue][color=green][color=darkred]
>>> @generatorize[/color][/color][/color]
.... def f():
.... return 1
....[color=blue][color=green][color=darkred]
>>> i = f()
>>> print i.next()[/color][/color][/color]
None[color=blue][color=green][color=darkred]
>>> print i.next()[/color][/color][/color]
None

Basically, I've assumed that 'do stuff' is the function that's being
wrapped.

Steve

[Peter Otten]

Re: creating generators from function

Simon Wittber wrote:
[color=blue]
> I use a coroutine/generator framework for simulating concurrent processes.
>
> To do this, I write all my functions using the general form:
>
> while True:
> do stuff
> yield None
>
> To make these generator functions compatible with a standard thread
> interface, I attempted to write a decorator which converts a standard
> function into a generator function (code attached below).
> Unfortunately, I received an exception, before I could test if my idea
> would work:
>
> TypeError: cannot create 'generator' instances
>
> I guess I can go the other way, and convert a generator into a
> function, but that just doesn't seem right.
>
> Is there anyway solve the problem described?[/color]

Generators are just functions with an extra flag set. I tried the following
modification of your code:

<code>
from opcode import opmap
globals().updat e(opmap)

def _f(): pass
function = type(_f)
del _f

def generatorize(f) :
co = f.func_code
nc = [ord(x) for x in co.co_code]
for i, op in enumerate(nc[:-1]):
if op == RETURN_VALUE:
nc[i] = YIELD_VALUE
newcode = ''.join([chr(x) for x in nc])

codeobj = type(co)(co.co_ argcount, co.co_nlocals, co.co_stacksize ,
co.co_flags + 32, newcode, co.co_consts,
co.co_names,
co.co_varnames, co.co_filename, co.co_name,
co.co_firstline no, co.co_lnotab, co.co_freevars,
co.co_cellvars)

return function(codeob j, f.func_globals, f.func_name,
f.func_defaults , f.func_closure)

@generatorize
def f():
while True:
return 1

def g():
while True:
yield 1

print g()
print f()

import itertools
print list(itertools. islice(f(), 5))
</code>

The hardest part was to find the enumerate() bug. I agree with Steven that
converting between functions and generators by replacing return with yield
and vice versa is hardly ever useful, though.

Peter

[Simon Wittber]

Re: creating generators from function

> I'm a little confused as to what you're trying to do here. The f()[color=blue]
> function, in particular, doesn't make much sense[/color]

I see I am misunderstood. The example was contrived, and it seems, incorrect.

I simulate threads, using generator functions and a scheduler.

My implementation lives here: http://metaplay.com.au/svn/LGT/LGT/nanothreads.py

If the same functions were to run inside a normal thread, they would
need to loop forever, rather than yield None on each iteration. I see
now that replacing return with yield won't work, a yield instruction
needs to be inserted somewhere.

I guess this changes my question to: Can I convert a function
containing an infinite loop, into a generator which will yield on each
iteration?

Sw.

[Timo Virkkala]

Re: creating generators from function

Simon Wittber wrote:[color=blue]
> I guess this changes my question to: Can I convert a function
> containing an infinite loop, into a generator which will yield on each
> iteration?[/color]

Why don't you just use threads? It would probably be easier in the long run...

--
Timo Virkkala

[Terry Reedy]

Re: creating generators from function


"Simon Wittber" <simonwittber@g mail.com> wrote in message
news:4e4a11f804 120806185e65f35 7@mail.gmail.co m...[color=blue]
> I guess this changes my question to: Can I convert a function
> containing an infinite loop, into a generator which will yield on each
> iteration?[/color]

A function containing an infinite loop will never return, so it is bugged
and useless unless it does has an external effect with something like
'print xxx' within the loop. Even then, it depends on 'while True:'
actually meaning 'while <external stop signal (control-C, reset, power off)
not present>'. The obvious change for a buggy function is to insert a
yield statement. For the implicit conditional with print, change 'print'
to 'yield'.

Note 1. Many algorithm books, especially when using pseudocode, use 'print
sequence_item' to avoid language specific details of list construction.
One of the beauties of Python2.2+ is that one can replace simply 'print'
with 'yield' and have a working algorithm.

Note 2. Delving deep into CPython internals, as you are, is CPython hacking
rather than Python programming per se. Fun, maybe, but I wonder if this is
really to best way to do what you want to do.

Terry J. Reedy

[Simon Wittber]

Re: creating generators from function

> A function containing an infinite loop will never return, so it is bugged[color=blue]
> and useless unless it does has an external effect with something like
> 'print xxx' within the loop.[/color]

I thought the idiom:

while True:
#code which iterates my simulation
if condition: break

wat quite normal. This is the style loop I am using.
[color=blue]
> Note 2. Delving deep into CPython internals, as you are, is CPython hacking
> rather than Python programming per se. Fun, maybe, but I wonder if this is
> really to best way to do what you want to do.[/color]

I am trying to allow two modes of concurrency, (threading, and
pseudo-threading using generators) without having to change my
application code. If either CPython or Python hacking can provide a
solution, I'm not sure, but I would be happy with either.

Sw.

[Terry Reedy]

Re: creating generators from function


"Simon Wittber" <simonwittber@g mail.com> wrote in message
news:4e4a11f804 1208171444a6c54 @mail.gmail.com ...[color=blue]
> I thought the idiom:
>
> while True:
> #code which iterates my simulation
> if condition: break
>
> wat quite normal. This is the style loop I am using.[/color]

Yes, quite normal. This is Python's version of do...until, synthesized
from while and break. As most people use the term, this is *not*
structurally an infinite loop (meaning no break) and also not functionally
so unless the condition is such that it never breaks either for some or all
inputs to the function.
[color=blue]
> I am trying to allow two modes of concurrency, (threading, and
> pseudo-threading using generators) without having to change my
> application code. If either CPython or Python hacking can provide a
> solution, I'm not sure, but I would be happy with either.[/color]

One Python level approach would be to quote the text of the function defs
that need to switch and then either programmaticall y edit them or not
before exec-ing them. Also messy, but more robust against version and
implementation changes.

Terry J. Reedy

[Mike Meyer]

Re: creating generators from function

Simon Wittber <simonwittber@g mail.com> writes:
[color=blue][color=green]
>> A function containing an infinite loop will never return, so it is bugged
>> and useless unless it does has an external effect with something like
>> 'print xxx' within the loop.[/color]
>
> I thought the idiom:
>
> while True:
> #code which iterates my simulation
> if condition: break
>
> wat quite normal. This is the style loop I am using.[/color]

I think it's a bit abnormal, because you have to scan the loop body
for breaks. I tend to write:

condition = True
while not condition:
#code which iterates my simulation

But that's just me.

<mike
--
Mike Meyer <mwm@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Isaac To]

Re: creating generators from function

>>>>> "Mike" == Mike Meyer <mwm@mired.or g> writes:

Mike> I think it's a bit abnormal, because you have to scan the
Mike> loop body for breaks. I tend to write:

Mike> condition = True
Mike> while condition: # corrected
Mike> #code which iterates my simulation

Then you'd have to scan the loop body to find the location where
condition is set, which is more difficult than locating breaks
normally. If you get a break, you really breaks. If you set
condition to False, you still might be modifying it to True later in
your code. And of course, most editors will highlight the "break" for
you, while no editor will highlight for you the "condition" variable
that you are staring at.

Regards,
Isaac.

[Mike Meyer]

Re: creating generators from function

Isaac To <iketo2@netscap e.net> writes:
[color=blue][color=green][color=darkred]
>>>>>> "Mike" == Mike Meyer <mwm@mired.or g> writes:[/color][/color]
>
> Mike> I think it's a bit abnormal, because you have to scan the
> Mike> loop body for breaks. I tend to write:
>
> Mike> condition = True
> Mike> while condition: # corrected
> Mike> #code which iterates my simulation
>
> Then you'd have to scan the loop body to find the location where
> condition is set, which is more difficult than locating breaks
> normally. If you get a break, you really breaks. If you set
> condition to False, you still might be modifying it to True later in
> your code. And of course, most editors will highlight the "break" for
> you, while no editor will highlight for you the "condition" variable
> that you are staring at.[/color]

Checking the condition and using a break at the bottom of the loop
doesn't change any of these things, so my version is less work. In
normal use, the "condition" is something that falls out of the
operation - checking for an empty string on file in put, for example
- and you can set the variable that is being tested to a value that
causes the condition to be true.

<mike
--
Mike Meyer <mwm@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.