yield_all needed in Python

[Terry Reedy]

Re: yield_all needed in Python


"Steven Bethard" <steven.bethard @gmail.com> wrote in message
news:aNSdnYFA69 ICiLjfRVn-ow@comcast.com. ..[color=blue]
> Douglas Alan wrote:[color=green]
>> In this case, that is great, since I'd much prefer
>>
>> yield *gen1(arg)
>>
>> than
>>
>> yield_all gen1(arg)[/color]
>
> I'm guessing the * syntax is pretty unlikely to win Guido's approval.
> There have been a number of requests[1][2][3] for syntax like:
>
> x, y, *rest = iterable
>
> for unpacking a variable sized list (where *rest would work in an
> analogous way to what it does in the args of a def.) Guido has
> consistently rejected these proposals, e.g.:
>
> "I think it's not worth adding now, and if you don't hear from me again
> on this topic it's because I haven't changed my mind..."
>
> My suspicion is that if he doesn't like the * syntax when there's a close
> parallel to the argument parsing usage, he's not likely to like it when
> there isn't one.[/color]

Hmm. My impression is that Guido did not like x,*y=iterable because he
does *not* see it as a 'close parallel' but as a strained analogy. To me,
yield *iterable is closer to the use in function calling. It would mean
'unpack in time' rather than 'unpack in space' but that is natural (to me,
anyway) since that is what generators are about.

In any case, I also like this better than yield_all and would estimate that
it a higher chance of accceptance, even if still under 50%. Part of the
justification could be that the abbreviated form is much easier to speed up
than the expanded form. If the ref count of the iterator is just 1, then
it might be reasonable to assume that iterator.next will not be called from
anywhere else. (IE, I can't think of an exception, but know that there
might be one somehow.)

Terry J. Reedy


Terry J. Reedy

[Steven Bethard]

Re: yield_all needed in Python

Terry Reedy wrote:[color=blue]
> "Steven Bethard" <steven.bethard @gmail.com> wrote in message
> news:aNSdnYFA69 ICiLjfRVn-ow@comcast.com. ..[color=green]
>>
>>My suspicion is that if he doesn't like the * syntax when there's a close
>>parallel to the argument parsing usage, he's not likely to like it when
>>there isn't one.[/color]
>
> Hmm. My impression is that Guido did not like x,*y=iterable because he
> does *not* see it as a 'close parallel' but as a strained analogy. To me,
> yield *iterable is closer to the use in function calling. It would mean
> 'unpack in time' rather than 'unpack in space' but that is natural (to me,
> anyway) since that is what generators are about.[/color]

I don't see the * in
yield *iterable
being much closer to the use in argument unpacking. Note what happens
to iterables after *:

py> def gen():
.... yield 1
.... yield 2
.... print "complete!"
....
py> def f(*args):
.... print args
....
py> f(*gen())
complete!
(1, 2)

The entire iterable is immediately exhausted. So if
yield *iterable
is supposed to parallel argument unpacking, I would expect that it would
also immediately exhaust the iterable, e.g. it would be equivalent to:
for item in tuple(iterable) :
yield item
which I don't think is what the OP wants.

I'm certain I could get used to the syntax. I'm only suggesting that I
don't find it very intuitive. (And I *have* thought a lot about
argument unpacking -- see my older threads about *args being an iterable
instead of a tuple.)

STeVe

[Douglas Alan]

Re: yield_all needed in Python

Nick Coghlan <ncoghlan@iinet .net.au> writes:
[color=blue]
> If you do write a PEP, try to get genexp syntax supported by the
> yield keyword.[/color]
[color=blue]
> That is, the following currently triggers a syntax error:
> def f():
> yield x for x in gen1(arg)[/color]

Wouldn't

yield *(x for x in gen1(arg))

be sufficient, and would already be supported by the proposal at
hand?

Also, with the syntax you suggest, it's not automatically clear
whether you want to yield the generator created by the generator
expression or the values yielded by the expression. The "*" makes
this much more explicit, if you ask me, without hindering readability.

|>oug

[Jeremy Bowers]

Re: yield_all needed in Python

On Wed, 02 Mar 2005 22:54:14 +1000, Nick Coghlan wrote:
[color=blue]
> Douglas Alan wrote:[color=green]
>> Steve Holden <steve@holdenwe b.com> writes:[color=darkred]
>>>Guido has generally observed a parsimony about the introduction of
>>>features such as the one you suggest into Python, and in particular
>>>he is reluctant to add new keywords - even in cases like decorators
>>>that cried out for a keyword rather than the ugly "@" syntax.[/color]
>>
>> In this case, that is great, since I'd much prefer
>>
>> yield *gen1(arg)[/color]
>
> If you do write a PEP, try to get genexp syntax supported by the yield keyword.
>
> That is, the following currently triggers a syntax error:
> def f():
> yield x for x in gen1(arg)[/color]

Hmmmm.

At first I liked this, but the reason that is a syntax error is that it is
"supposed" to be

def f():
yield (x for x in gen1(arg))

which today on 2.4 returns a generator instance which will in turn
yield one generator instance from the genexp, and I am quite uncomfortable
with the difference between the proposed behaviors with and without the
parens.

Which sucks, because at first I really liked it :-)

We still would need some syntax to say "yield this 'in place' rather than
as an object".

Moreover, since "yield" is supposed to be analogous to "return", what does

return x for x in gen1(arg)

do? Both "it returns a list" and "it returns a generator" have some
arguments in their favor.

And I just now note that any * syntax, indeed, any syntax at all will
break this.

You know, given the marginal gains this gives anyway, maybe it's best off
to just observe that in the event that this is really important, it's
possible to hand-code the short-circuiting without too much work, and let
people write a recipe or something.

def genwrap(*genera tors):
while generators:
try:
returnedValue = generators[-1].next()
if hasattr(returne dValue, 'next'):
generators.appe nd(returnedValu e)
continue
yield returnedValue
except StopIteration:
generators.pop( )

Not tested at all because the wife is calling and I gotta go :-)

[Skip Montanaro]

Re: yield_all needed in Python


Jeremy> At first I liked this, but the reason that is a syntax error is
Jeremy> that it is "supposed" to be

Jeremy> def f():
Jeremy> yield (x for x in gen1(arg))

Jeremy> which today on 2.4 returns a generator instance which will in
Jeremy> turn yield one generator instance from the genexp, and I am
Jeremy> quite uncomfortable with the difference between the proposed
Jeremy> behaviors with and without the parens.

Jeremy> Which sucks, because at first I really liked it :-)

Jeremy> We still would need some syntax to say "yield this 'in place'
Jeremy> rather than as an object".

def f():
yield from (x for x in gen1(arg))

Skip

[Francis Girard]

Re: yield_all needed in Python

Le mercredi 2 Mars 2005 21:32, Skip Montanaro a écrit :[color=blue]
> def f():
>     yield from (x for x in gen1(arg))
>
> Skip[/color]

This suggestion had been made in a previous posting and it has my preference :

def f():
yield from gen1(arg)

Regards

Francis

[Nick Coghlan]

Re: yield_all needed in Python

Douglas Alan wrote:[color=blue]
> Wouldn't
>
> yield *(x for x in gen1(arg))
>
> be sufficient, and would already be supported by the proposal at
> hand?[/color]

It would, but, as Steven pointed out, the * in func(*args) results in
tuple(args) being passed to the underlying function.

So I see no reason to expect "yield *iterable" to imply a for loop that yields
the iterators contents. IMO, it's even more of a stretch than the tuple
unpacking concept (at least that idea involves tuples!)

Whereas:

yield x for x in iterable if condition

Maps to:
for x in iterable:
if condition:
yield x

Just as:
[x for x in iterable if condition]

Maps to:
lc = []
for x in iterable:
if condition:
lc.append(x)

And:
(x for x in iterable if condition)

Maps to:
def g()
for x in iterable:
if condition:
yield x

And removing a couple of parentheses is at least as clear as adding an asterisk
to the front :)

Cheers,
Nick.

--
Nick Coghlan | ncoghlan@email. com | Brisbane, Australia
---------------------------------------------------------------

http://boredomandlaziness.skystorm.net

[Nick Coghlan]

Re: yield_all needed in Python

Jeremy Bowers wrote:[color=blue]
> At first I liked this, but the reason that is a syntax error is that it is
> "supposed" to be
>
> def f():
> yield (x for x in gen1(arg))
>
> which today on 2.4 returns a generator instance which will in turn
> yield one generator instance from the genexp[/color]

And it would continue to do so in the future. On the other hand, removing the
parens makes it easy to write things like tree traversal algorithms:

def left_to_right_t raverse(node):
yield x for x in node.left
yield node .value
yield x for x in node.right

In reality, I expect yielding each item of a sub-iterable to be more common than
building a generator that yields generators.
[color=blue]
> , and I am quite uncomfortable
> with the difference between the proposed behaviors with and without the
> parens.[/color]

Why? Adding parentheses can be expected to have significant effects when it
causes things to be parsed differently. Like the example I posted originally:

[x for x in iterable] # List comp (no parens == eval in place)
[(x for x in iterable)] # Parens - generator goes in list

Or, for some other cases where parentheses severely affect parsing:

print x, y
print (x, y)

assert x, y
assert (x, y)

If we want to pass an iterator into a function, we use a generator expression,
not extended call syntax. It makes sense to base a sub-iterable yield syntax on
the former, rather than the latter.
[color=blue]
> Moreover, since "yield" is supposed to be analogous to "return", what does
>
> return x for x in gen1(arg)
>
> do? Both "it returns a list" and "it returns a generator" have some
> arguments in their favor.[/color]

No, it would translate to:

for x in gen1(arg):
return x

Which is nonsense, so you would never make it legal.
[color=blue]
> And I just now note that any * syntax, indeed, any syntax at all will
> break this.[/color]

As you noted, this argument is specious because it applies to *any* change to
the yield syntax - yield and return are fundamentally different, since yield
allows resumption of processing on the next call to next().
[color=blue]
> You know, given the marginal gains this gives anyway,[/color]

I'm not so sure the gains will be marginal. Given the penalties CPython imposes
on recursive calls, eliminating the nested "next()" invocations could
significantly benefit any code that uses nested iterators.

An interesting example where this could apply is:

def flatten(iterabl e):
for item in iterable:
if item is iterable:
# Do the right thing for self-iterative things
# like length 1 strings
yield iterable
raise StopIteration
try:
itr = iter(item):
except TypeError:
yield item
else:
yield x for x in flatten(item)

Cheers,
Nick.

P.S. Which looks more like executable pseudocode?

def traverse(node):
yield *node.left
yield node .value
yield *node.right

def traverse(node):
yield x for x in node.left
yield node .value
yield x for x in node.right

--
Nick Coghlan | ncoghlan@email. com | Brisbane, Australia
---------------------------------------------------------------

http://boredomandlaziness.skystorm.net

[Paul Moore]

Re: yield_all needed in Python

Skip Montanaro <skip@pobox.com > writes:
[color=blue]
> Doug> def foogen(arg1):
>
> Doug> def foogen1(arg2):
> Doug> # Some code here
>
> Doug> # Some code here
> Doug> yield_all foogen1(arg3)
> Doug> # Some code here
> Doug> yield_all foogen1(arg4)
> Doug> # Some code here
> Doug> yield_all foogen1(arg5)
> Doug> # Some code here
> Doug> yield_all foogen1(arg6)
>
> If this idea advances I'd rather see extra syntactic sugar introduced to
> complement the current yield statement instead of adding a new keyword.[/color]

You can work around the need for something like yield_all, or
explicit loops, by defining an "iflatten" generator, which yields
every element of its (iterable) argument, unless the element is a
generator, in which case we recurse into it:
[color=blue][color=green][color=darkred]
>>> from types import GeneratorType
>>> def iflatten(it):[/color][/color][/color]
.... it = iter(it)
.... for val in it:
.... if isinstance(val, GeneratorType):
.... for v2 in iflatten(val):
.... yield v2
.... else:
.... yield val

To take this one step further, you can define an @iflattened
decorator (yes, it needs a better name...)
[color=blue][color=green][color=darkred]
>>> def iflattened(f):[/color][/color][/color]
.... def wrapper(*args, **kw):
.... for val in iflatten(f(*arg s, **kw)):
.... yield val
.... return wrapper

Now, we can do things like:
[color=blue][color=green][color=darkred]
>>> @iflattened[/color][/color][/color]
.... def t():
.... def g1():
.... yield 'a'
.... yield 'b'
.... yield 'c'
.... def g2():
.... yield 'd'
.... yield 'e'
.... yield 'f'
.... yield g1()
.... yield 1
.... yield g2()
....[color=blue][color=green][color=darkred]
>>> list(t())[/color][/color][/color]
['a', 'b', 'c', 1, 'd', 'e', 'f']

This can probably be tidied up and improved, but it may be a
reasonable workaround for something like the original example.

Paul.
--
The most effective way to get information from usenet is not to ask
a question; it is to post incorrect information. -- Aahz's Law

[Jeremy Bowers]

Re: yield_all needed in Python

On Thu, 03 Mar 2005 20:47:42 +0000, Paul Moore wrote:[color=blue]
> This can probably be tidied up and improved, but it may be a
> reasonable workaround for something like the original example.[/color]

This is why even though in some sense I'd love to see yield *expr, I can't
imagine it's going to get into the language itself; it's too easy to do it
yourself, or provide a library function to do it (which would A: Be a lot
easier if we had some sort of "iterable" interface support and B: Be a
great demonstration of something useful that really needs protocol support
to come off right, because isinstance(some thing, GeneratorType) isn't
sufficient in general).

Abstractly I like the star syntax, but concretely I'm not a big fan of
adding something to the language that can be done right now with a fairly
short function/generator and hardly even any extra keystrokes to invoke it
when done right, and that overrides my abstract appreciation.

[Isaac To]

Re: yield_all needed in Python

>>>>> "Paul" == Paul Moore <pf_moore@yahoo .co.uk> writes:

Paul> You can work around the need for something like yield_all,
Paul> or explicit loops, by defining an "iflatten" generator,
Paul> which yields every element of its (iterable) argument,
Paul> unless the element is a generator, in which case we recurse
Paul> into it:
Paul> ...

Only if you'd never want to yield a generator.

Regards,
Isaac.