PEP 322: Reverse Iteration (REVISED, please comment)

[Raymond Hettinger]

Based on your extensive feedback, PEP 322 has been completely revised.
The response was strongly positive, but almost everyone preferred
having a function instead of multiple object methods. The updated
proposal is at:

https://www.python.org/peps/pep-0322.html

In a nutshell, it proposes a builtin function that greatly simplifies reverse
iteration. The core concept is that clarity comes from specifying a
sequence in a forward direction and then saying, "inreverse( )":

for elem in inreverse(seqn) :
. . .

Unlike seqn[::-1], this produces a fast iterator instead of a full reversed
copy.

Discussions with Guido made it clear that inreverse() will not be extended
to cover all iterables. The proposal is about simplicity, expression, and
performance. As such, it would be counter-productive to take in a general
iterable, run it to completion, save the data in memory, and then iterate
over the data in reverse.


Raymond Hettinger

[Werner Schiendl]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

Raymond Hettinger wrote:
[color=blue]
>
> Discussions with Guido made it clear that inreverse() will not be extended
> to cover all iterables. The proposal is about simplicity, expression, and
> performance. As such, it would be counter-productive to take in a general
> iterable, run it to completion, save the data in memory, and then iterate
> over the data in reverse.
>[/color]

Which of course can easily be done explicitely:

# Make some iterable unsuitable for inreverse
a = iter("Hello")

# Create a list that can be in'reversed explicitely
for i in inreverse(list( a)):
print i


As always, explicit is better than implicit ;-)

Maybe something alike should go into the docs...


- Werner

[Werner Schiendl]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

Hi,

was it considered to name the function just "reverse"?

This would read very naturally to me:

for elem in reverse(seq):
print elem


Also if you use it in function calls it looks good to me:

x = dostuff(reverse (seq), a, b)


Another possibility would be to have a static method for iter:

for elem in iter.reversed(s ed):
print elem

(in that case I'd use an attribute rather than a verb, because
the generated iter(ator) will be reverse_d_).

The static method approach would clearly document that the
result is an iterator (which none of the other names proposed
really does IMHO)


In general, I'm +1 on the PEP


regards

Werner

[Paul Moore]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

python@rcn.com (Raymond Hettinger) writes:
[color=blue]
> In a nutshell, it proposes a builtin function that greatly simplifies reverse
> iteration. The core concept is that clarity comes from specifying a
> sequence in a forward direction and then saying, "inreverse( )":
>
> for elem in inreverse(seqn) :
> . . .[/color]

I've got to say, I really don't like the name. Particularly, the
repeated "in" in "in inreverse..." strikes me as ugly.
[color=blue]
> Discussions with Guido made it clear that inreverse() will not be extended
> to cover all iterables. The proposal is about simplicity, expression, and
> performance. As such, it would be counter-productive to take in a general
> iterable, run it to completion, save the data in memory, and then iterate
> over the data in reverse.[/color]

I like the proposal, it's simple and expressive. But please, change
the name! I assume there's a good reason why the obvious "reverse()"
wasn't chosen. Of the alternatives in the PEP, backwards() doesn't
seem to read nicely (but it's better than inreverse!!!), and ireverse
is OK, but makes me think that it should be in itertools (or all the
itertools should be builtin). Some other possibilities:

rev(seq) - probably too abbreviated
reversed(seq) - adjective rather than verb, but reads well

As usual, the semantics is perfectly acceptable, but we'll have a nice
long argument over the name :-)

Paul.
--
This signature intentionally left blank

[Alex Martelli]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

Werner Schiendl wrote:
...[color=blue]
> Another possibility would be to have a static method for iter:
>
> for elem in iter.reversed(s ed):
> print elem
>
> (in that case I'd use an attribute rather than a verb, because
> the generated iter(ator) will be reverse_d_).
>
> The static method approach would clearly document that the
> result is an iterator (which none of the other names proposed
> really does IMHO)[/color]

I like this one! Another advantage is, one fewer built-in.

Problem is, iter is currently a built-in function, not a type...


Alex

[John Roth]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

+1


"Raymond Hettinger" <python@rcn.com > wrote in message
news:5d83790c.0 310281022.6264a a16@posting.goo gle.com...[color=blue]
> Based on your extensive feedback, PEP 322 has been completely revised.
> The response was strongly positive, but almost everyone preferred
> having a function instead of multiple object methods. The updated
> proposal is at:
>
> www.python.org/peps/pep-0322.html
>
> In a nutshell, it proposes a builtin function that greatly simplifies[/color]
reverse[color=blue]
> iteration. The core concept is that clarity comes from specifying a
> sequence in a forward direction and then saying, "inreverse( )":
>
> for elem in inreverse(seqn) :
> . . .
>
> Unlike seqn[::-1], this produces a fast iterator instead of a full[/color]
reversed[color=blue]
> copy.
>
> Discussions with Guido made it clear that inreverse() will not be extended
> to cover all iterables. The proposal is about simplicity, expression, and
> performance. As such, it would be counter-productive to take in a general
> iterable, run it to completion, save the data in memory, and then iterate
> over the data in reverse.[/color]

It's certainly clear enough, and I like it in general.

I'd appreciate a bit of discussion about why reverse() was rejected as
the name, though.

John Roth[color=blue]
>
>
> Raymond Hettinger[/color]

[Dave Benjamin]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

In article <5d83790c.03102 81022.6264aa16@ posting.google. com>, Raymond Hettinger wrote:[color=blue]
> for elem in inreverse(seqn) :
> . . .[/color]

Is there an echo in inhere? +1 +1otherwise... =)

--
..:[ dave benjamin (ramenboy) -:- www.ramenfest.com -:- www.3dex.com ]:.
: d r i n k i n g l i f e o u t o f t h e c o n t a i n e r :

[Patrick Maupin]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

Raymond Hettinger wrote:
[color=blue]
> Based on your extensive feedback, PEP 322 has been completely revised.[/color]

Personally, for some tasks I have used reverse iteration quite a bit.

My use case has _always_ been that I want to modify a list (e.g.
adding or deleting items) in situ. In all other cases which I can
recall, I have quite happily iterated in the forward direction.

(Maybe this is because I am not afraid to call list.reverse(), and
insertion/deletion is the only use case where this doesn't really help.)

This means that the implementation restrictions described in the PEP are
fine with me (because a list has the requisite methods).

However, this also means that I am most interested in how the enumerate()
interface will work with this (because the current list index must be
known in order to properly add or delete items). enumerate() was
mentioned in the PEP but not discussed in any real depth.

My personal preference (since this is the only use case which I have
ever encountered) would be to enhance enumerate in a reasonable fashion,
and not necessarily even have (or at least care about) the reverse/ireverse/
whatever name underneath enumerate().

I will happily accept whatever syntax comes along (unless the enumerate
counts 0,1,2,3... while the index of the adjacent list object is decremented
[-1],[-2],[-3]... :)

BUT, for sheer usability I think it would be wonderful if enumerate()
took an optional second argument to let it know you want to go backward.
The most general syntax for this second argument would naturally be
a slice:

for idx,obj in enumerate(mylis t,-1:-1:-1):
do whatever

but if people think this is too ugly something simpler could be
substituted (but think about the useful possibilities of a slice
argument here before you complain too hard about the ugliness...)

Some may argue that this restricts enumerate() to certain kinds of
iterators when using the slice (and thus adds a lot of special casing),
but it may be that, realistically, if ireverse() is going to have this
special casing, and enumerate() is going to work with ireverse(), you
already have the same problem (unless the user is expected to work
with the enumerate counting backwards from the actual list index).
I cannot say this for sure, however, because I did not see (or perhaps
read closely enough) the part of the PEP which discussed enumerate.

Regards,
Pat

[Werner Schiendl]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

Alex Martelli wrote:[color=blue]
> Werner Schiendl wrote:
> ...[color=green]
>>Another possibility would be to have a static method for iter:
>>
>>for elem in iter.reversed(s ed):
>> print elem
>>[/color]
>
> I like this one! Another advantage is, one fewer built-in.
>
> Problem is, iter is currently a built-in function, not a type...
>[/color]

An additional advantage that came to my mind is that it would fit
Guido's preference about how to (possible) implement inline sort.

having

L = list.sorted(seq )

i = iter.reversed(s eq)

seems very consistent to me.


best regards

Werner

[Raymond Hettinger]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

> The static method approach would clearly document that the[color=blue]
> result is an iterator (which none of the other names proposed
> really does IMHO)[/color]

Better name are welcome!

Moving it somewhere else is not open. It is proposed as a builtin for a
reason -- it is a core looping tool like zip() or enumerate() and it is meant
to simplify rather than complicate code.

Static methods, class methods, and weird descriptors be darned, this is
not an exercise in how weirdly it can be implemented just to avoid
having a builtin.

Would everything be somehow better if Alex's wonderful sum() had
been implemented as a int.sum() classmethod or was tucked way in
another module? Of course not! Likewise, would zip() or enumerate()
be successful solutions to lock-step iteration and the loop-counter
problems if they were iter.zip() and iter.enumerate( )? No, of course not.
Let's put an end to this silliness right now. The idea is offered as a
builtin or not at all; otherwise, the existing [::-1] starts to look better.

[color=blue]
> In general, I'm +1 on the PEP[/color]

Thanks! I certain it will make code more readable and reviewable.

The challenge with a PEP this simple is that experts feel this
overpowering urge to apply all their know-how and transform
in to something other than a clean, fast, simple solution.


Raymond Hettinger

[Terry Reedy]

Re: PEP 322: Reverse Iteration (REVISED, please comment)


"Werner Schiendl" <n17999950.temp .werner@neverbo x.com> wrote in
message news:3f9f9d2a$1 @brateggebdc5.b r-automation.co.a t...[color=blue][color=green]
> > Problem is, iter is currently a built-in function, not a type...[/color][/color]

Since iter() constructs and returns a type 'iterator' object, I
expected that it might be a type object, just like int(), etc. If it
were turned into one, like int(), etc, have been, then
[color=blue]
> L = list.sorted(seq )
> i = iter.reversed(s eq)
> seems very consistent to me.[/color]

would be exactly parallel and hence very consistent.

Unless there is a good reason to not make iter a type object, then
making it so could be part of the suggested implementation of the PEP.

Terry J. Reedy

[Raymond Hettinger]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

> Since iter() constructs and returns a type 'iterator' object, I[color=blue]
> expected that it might be a type object, just like int(), etc. If it
> were turned into one, like int(), etc, have been, then[/color]
. . .[color=blue]
> Unless there is a good reason to not make iter a type object, then
> making it so could be part of the suggested implementation of the PEP.[/color]

iter() is a factory function that can return all kinds of things:
[color=blue][color=green][color=darkred]
>>> from random import random
>>> iters = iter('str'), iter(['list']), iter(dict(a=1)) , iter(random, None)
>>> map(type, iters)[/color][/color][/color]
[<type 'iterator'>, <type 'listiterator'> , <type 'dictionary-iterator'>, <type
'callable-iterator'>]



Let's see if we can get back to the merits of the pep.
Looking at your own code, can you verify that ireverse()
is an improvement over what you have now.


Raymond Hettinger

[Werner Schiendl]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

Raymond Hettinger wrote:
[color=blue]
>
> Better name are welcome!
>
> Moving it somewhere else is not open. It is proposed as a builtin for a
> reason -- it is a core looping tool like zip() or enumerate() and it is meant
> to simplify rather than complicate code.
>[/color]

Well, that of course limits the choice :-)

Let's see what's available:

inreverse --

seems a little clumsy when used in a "for item in inreverse(seq): ",
which is probably the place where it's used most often.

The term itself seems reasonable to me.

ireverse --

IMHO that would suggest it belonged into itertools - which it
according to your explanatation does NOT.

So I'd rather 'reserve' that term if one day something there
is needed (or end up with iireverse ;-)

To my non-native-english eyes it's also a bit ugly.

And concerning simplicity, how to explain a newby the "i"
and why it's with "reverse" but not with "enumerate" .

reverse --

According to the PEP this is not for discussion, although I'm
with John Roth in that I think this were worth some more
discussion.

I do not use Python too much (yet as often as possible, but
"job work" is done with something else.
Still I cannot imagine why anyone should confuse a method of
a list object (that does return None) with a builtin function
that returns "something" to walk through an arbitrary sequence
in reverse.

This one fits IMHO also best to the most-direct relatives you
mention in your post (zip, enumerate) in that it is a verb
without any prefix.

backwards --

I cannot remember having seem that term in any library or
programming language, it feels strange to me.

But it's still correct, a verb, fits the style IMHO.
Like it more than ireverse.


So my order from most-prefered to least-prefered is:

reverse

inreverse

backwards

ireverse
[color=blue]
>
> The challenge with a PEP this simple is that experts feel this
> overpowering urge to apply all their know-how and transform
> in to something other than a clean, fast, simple solution.
>[/color]

I'd not actually claim myself a Python expert ;-)


best regards

Werner

[Peter Otten]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

Raymond Hettinger wrote:
[color=blue][color=green]
>> Since iter() constructs and returns a type 'iterator' object, I
>> expected that it might be a type object, just like int(), etc. If it
>> were turned into one, like int(), etc, have been, then[/color]
> . . .[color=green]
>> Unless there is a good reason to not make iter a type object, then
>> making it so could be part of the suggested implementation of the PEP.[/color]
>
> iter() is a factory function that can return all kinds of things:
>[color=green][color=darkred]
>>>> from random import random
>>>> iters = iter('str'), iter(['list']), iter(dict(a=1)) , iter(random,
>>>> None) map(type, iters)[/color][/color]
> [<type 'iterator'>, <type 'listiterator'> , <type 'dictionary-iterator'>,
> [<type
> 'callable-iterator'>][/color]

So why couldn't all these iterators inherit from iter just as unicode and
str do from basestring?
[color=blue]
> Let's see if we can get back to the merits of the pep.[/color]

I respect that you want to keep this discussion focused, and itertools is my
favourite new package - but sometimes il faut reculer pour mieux sauter :-)
[color=blue]
> Looking at your own code, can you verify that ireverse()
> is an improvement over what you have now.[/color]

While I'm zipping and enumerating all the time, reversing is rare, so I'm
not desperately awaiting this builtin.
The most common usecase seems iteration over a sequence that is mutated in
the process:

class mutate(object):
def __init__(self, alist):
self.index = -1
self.alist = alist
def next(self):
self.index += 1
try:
self.alist[self.index]
except IndexError:
raise StopIteration
return self
def value(self):
return self.alist[self.index]
def __iter__(self):
return self
def delete(self):
del self.alist[self.index]
self.index -= 1

sample = range(10)
for item in mutate(sample):
if item.value() in [3,5,7]:
item.delete()
print sample

I'm sure that the above approach can be improved (in particular, it must not
rely on an index, i. e. random access) as I'm sure that it makes the
coder's intention clearer than the "reverse to enable deletion" idiom.

Peter

[Alex Martelli]

Re: PEP 322: Reverse Iteration (REVISED, please comment)

Peter Otten wrote:
[color=blue]
> Raymond Hettinger wrote:[/color]
,,,[color=blue][color=green]
>> iter() is a factory function that can return all kinds of things:[/color][/color]

Right, and thus it must stay. Oh well.
[color=blue]
> So why couldn't all these iterators inherit from iter just as unicode and
> str do from basestring?[/color]

They might, but then calling iter() should complain that iter is an
abstract baseclass, not instantiable, just like calling basestring()
does. Having an abstract baseclass that DOES return new objects when
called would be very perverse indeed. So, making iter a type is not
really a sensible option, alas.


Alex