Comment on PEP-0322: Reverse Iteration Methods

[Raymond Hettinger]

Please comment on the new PEP for reverse iteration methods.
Basically, the idea looks like this:

for i in xrange(10).iter _backwards(): # 9,8,7,6,5,4,3,2 ,1,0
<do something with i>

The HTML version is much more readable than the ReST version.
See:

http://www.python.org/peps/pep-0322.html

Several interesting ideas surfaced in the pre-pep thread:

* Call it ireverse() instead of iter_backwards( ).

Good idea. This is much more pithy.


* Use object properties instead of methods.

I can't yet claim to understand what the author is really
proposing. It has something to do which providing
access to an object that responds to iter, getitem, and
getslice with reversed indices.


* using a single function that looks for an __riter__ magic
method and, if not found, use __getitem__ and __len__
to build a reverse iterator.

A workable version of this was posted.
It saves implementing some object methods at the
expense of adding a new builtin function and of
creating a new magic method name.
It is slower than direct access to the underlying object.
It crashes when applied to an infinite iterator.
It produces bizarre results when applied to mappings.


* special markers for infinite iterators

This idea is interesting but doesn't extend well
when the object is wrapped by another iterator.
Also, there is no automatic way to determine
which generators can be infinite.


Raymond Hettinger

[Sean Ross]

Re: Comment on PEP-0322: Reverse Iteration Methods

"Raymond Hettinger" <vze4rx4y@veriz on.net> wrote in message
news:95rcb.2002 $yU5.1775@nwrdn y01.gnilink.net ...[color=blue]
> Please comment on the new PEP for reverse iteration methods.[/color]

# from PEP 322
"""
Rejected Alternative Ideas
[snip]
Add a builtin function, reverse() ...
"""

Do you mean add a function reverse() (or ireverse()) to the itertools
library, or to __builtins__?

[John Roth]

Re: Comment on PEP-0322: Reverse Iteration Methods

I think it says what needs to be said.

+1

John Roth

"Raymond Hettinger" <vze4rx4y@veriz on.net> wrote in message
news:95rcb.2002 $yU5.1775@nwrdn y01.gnilink.net ...[color=blue]
> Please comment on the new PEP for reverse iteration methods.
> Basically, the idea looks like this:
>
> for i in xrange(10).iter _backwards(): # 9,8,7,6,5,4,3,2 ,1,0
> <do something with i>
>
> The HTML version is much more readable than the ReST version.
> See:
> http://www.python.org/peps/pep-0322.html
>
>
> Several interesting ideas surfaced in the pre-pep thread:
>
> * Call it ireverse() instead of iter_backwards( ).
>
> Good idea. This is much more pithy.
>
>
> * Use object properties instead of methods.
>
> I can't yet claim to understand what the author is really
> proposing. It has something to do which providing
> access to an object that responds to iter, getitem, and
> getslice with reversed indices.
>
>
> * using a single function that looks for an __riter__ magic
> method and, if not found, use __getitem__ and __len__
> to build a reverse iterator.
>
> A workable version of this was posted.
> It saves implementing some object methods at the
> expense of adding a new builtin function and of
> creating a new magic method name.
> It is slower than direct access to the underlying object.
> It crashes when applied to an infinite iterator.
> It produces bizarre results when applied to mappings.
>
>
> * special markers for infinite iterators
>
> This idea is interesting but doesn't extend well
> when the object is wrapped by another iterator.
> Also, there is no automatic way to determine
> which generators can be infinite.
>
>
> Raymond Hettinger
>
>[/color]

[Stephen Horne]

Re: Comment on PEP-0322: Reverse Iteration Methods

On Thu, 25 Sep 2003 00:58:45 GMT, "Raymond Hettinger"
<vze4rx4y@veriz on.net> wrote:
[color=blue]
>* Use object properties instead of methods.
>
> I can't yet claim to understand what the author is really
> proposing. It has something to do which providing
> access to an object that responds to iter, getitem, and
> getslice with reversed indices.[/color]

The idea is basically to have a way to get a proxy to a sequence which
behaves like the original sequence, but with indexes reversed. There
is no reason why member functions couldn't return those proxies
instead of using properties, except that when the result is sliced it
looks less ugly.

I have a demo of it for lists only (listing at end of post). It has
three properties...

backward : reversed, slicing returns a list
ibackward : reversed, slicing returns an iterator
iforward : forward, slicing returns an iterator

All properties currently support __getitem__, __setitem__,
__delitem__, __str__, __repr__, __iter__ and __len__ (they all return
an instance of the same proxy class but with different options set).

For simple iterating-over-ints, I'd use the same principles with that
object-acting-as-sliceable-set-of-all-integers idea from the PEP284
thread.


I had some problems getting the slice translation to work - there are
some little nasties with negative steps. You can write...

x [::-1]

....but trying to make it explicit...

x [len(x)-1:-1:-1]

....breaks it because of that -ve stop bound. Not being a Numeric user,
slice steps are quite a new thing to me - and that little trap just
caught me by surprise. It isn't too hard to work around it, though.
The code following isn't pretty but so far as I can tell it works.

Python 2.3 is assumed.

The question is - is the exercise worthwhile. I'm not sure even though
it's my own idea. I think it is nice in its way, and a lot more
flexible than the 'iter_backward' method proposal, but that is quite
likely a bad thing as it is probably massive overkill for the problem
being solved - and in particular the extra flexibility has costs.

What the hell, though - it was fun to play with for a bit!

Anyway, here is a quick test session...

[color=blue][color=green][color=darkred]
>>> from rilist import rilist
>>> x=rilist(range( 20))
>>> x.backward[/color][/color][/color]
[19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0][color=blue][color=green][color=darkred]
>>> x.backward[:10][/color][/color][/color]
[19, 18, 17, 16, 15, 14, 13, 12, 11, 10][color=blue][color=green][color=darkred]
>>> x.backward[::2][/color][/color][/color]
[19, 17, 15, 13, 11, 9, 7, 5, 3, 1][color=blue][color=green][color=darkred]
>>> del x.backward[::2]
>>> x.backward[/color][/color][/color]
[18, 16, 14, 12, 10, 8, 6, 4, 2, 0][color=blue][color=green][color=darkred]
>>> x[/color][/color][/color]
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18][color=blue][color=green][color=darkred]
>>> for i in x.backward :[/color][/color][/color]
.... print i
....
18
16
14
12
10
8
6
4
2
0[color=blue][color=green][color=darkred]
>>> for i in x.backward [5:]:[/color][/color][/color]
.... print i
....
8
6
4
2
0[color=blue][color=green][color=darkred]
>>> for i in x [5:]:[/color][/color][/color]
.... print i
....
10
12
14
16
18


I've done some 'ibackward' tests as well, and that seems to work - but
I haven't bothered with 'iforward'.


Here is the source - warning, it isn't very self-documenting ATM...


class c_List_Proxy (object) :
"""
Proxy to list class which supports a subset of methods with
modified behaviours, mainly...

- Slicing may create an iterator rather than a list.
- Subscripts and ranges may be reversed.
"""

def __init__ (self, p_List, p_Reverse=False , p_Iter_Slice=Fa lse) :
self.List = p_List
self.Reverse = p_Reverse
self.Iter_Slice = p_Iter_Slice

# need some support stuff

def __SliceIter__ (self, p_Slice) :
if p_Slice.step > 0 :
i = p_Slice.start
while i < p_Slice.stop :
yield self.List [i]
i += p_Slice.step
else :
i = p_Slice.start
while i > p_Slice.stop :
yield self.List [i]
i += p_Slice.step

def __FullIter__ (self) :
i = len (self.List) - 1
while i >= 0 :
yield self.List [i]
i -= 1

def __KeyTransforme d__ (self, p_Key) :
if self.Reverse :
if p_Key < 0 :
return (-1) - p_Key

else :
return (len (self.List) - 1) - p_Key

else :
return p_Key

def __Bound__ (self, p_Given, p_Default) :
if p_Given is None :
return p_Default
elif p_Given < 0 :
return len (self.List) + p_Given
else :
return p_Given

def __SliceFixed__ (self, p_Slice) :
l_Step = p_Slice.step or 1

if l_Step == 0 :
raise IndexError, "Step must be non-zero"

elif l_Step > 0 :
l_Start = self.__Bound__ (p_Slice.start, 0)
l_Stop = self.__Bound__ (p_Slice.stop, len (self.List))

else :
l_Start = self.__Bound__ (p_Slice.start, len (self.List) - 1)
l_Stop = self.__Bound__ (p_Slice.stop, -1)

l_Count = (((l_Stop - 1) - l_Start) // l_Step) + 1
l_Stop = l_Start + l_Count * l_Step

return slice(l_Start,l _Stop,l_Step)

def __SliceTransfor med__ (self, p_Slice) :
l_Slice = self.__SliceFix ed__ (p_Slice)

if self.Reverse :
l_End = len(self.List) - 1
return slice (l_End - l_Slice.start, l_End - l_Slice.stop,
-l_Slice.step)

else :
return p_Slice

# some members are trivial

def __len__ (self) :
return len (self.List)

def __iter__ (self) :
return self.__FullIter __ ()

# some members need a bit more work...

def __str__ (self) :
if self.Reverse :
return str(self.List [::-1])
else :
return str(self.List)

def __repr__ (self) :
if self.Reverse :
return repr(self.List [::-1])
else :
return repr(self.List)

def __getitem__ (self, p_Item) :
if isinstance (p_Item, slice) :
if self.Iter_Slice :
return self.__SliceIte r__ (self.__SliceTr ansformed__ (p_Item))

else :
l_Slice = self.__SliceTra nsformed__ (p_Item)

if l_Slice.stop == -1 : # annoying special case
return self.List [l_Slice.start:: l_Slice.step]

return self.List [l_Slice.start:l _Slice.stop:l_S lice.step]

else :
return self.List [self.__KeyTrans formed__ (p_Item)]

def __setitem__ (self, p_Item, p_Value) :
if isinstance (p_Item, slice) :
l_Slice = self.__SliceTra nsformed__ (p_Item)

if l_Slice.stop == -1 : # annoying special case
self.List [l_Slice.start:: l_Slice.step] = p_Value
else :
self.List [l_Slice.start:l _Slice.stop:l_S lice.step] = p_Value

else :
self.List [self.__KeyTrans formed__ (p_Item)] = p_Value

def __delitem__ (self, p_Item) :
if isinstance (p_Item, slice) :
l_Slice = self.__SliceTra nsformed__ (p_Item)

if l_Slice.stop == -1 : # annoying special case
del self.List [l_Slice.start:: l_Slice.step]
else :
del self.List [l_Slice.start:l _Slice.stop:l_S lice.step]

else :
del self.List [self.__KeyTrans formed__ (p_Item)]

class rilist (list) :
"""
Extend normal list to provide properties giving variants of normal
indexing behaviour - specifically...

indexing slicing returns
-------- ---------------
iforward forward iterator
ibackward backward iterator
backward backward list
"""
def __IFwd__ (self) :
return c_List_Proxy (self, False, True)

def __IBack__ (self) :
return c_List_Proxy (self, True, True)

def __Back__ (self) :
return c_List_Proxy (self, True, False)

iforward = property(__IFwd __ )
ibackward = property(__IBac k__)
backward = property(__Back __ )


--
Steve Horne

steve at ninereeds dot fsnet dot co dot uk

[Andrew Dalke]

Re: Comment on PEP-0322: Reverse Iteration Methods

Raymond Hettinger:[color=blue]
> Please comment on the new PEP for reverse iteration methods.[/color]

It says:
] Also, it only works with lists (strings, for example, do not define a
reverse method):
]
] rseqn = list(seqn)
] rseqn.reverse()
] for value in rseqn:
] print value

However, the example code will work on a string, because list(string)
returns a list, which does have a reverse.

] Extending slicing minimizes the code overhead but does nothing
] for memory efficiency, beauty, or clarity.

I don't agree with all three of those. Take for example your mhlib
use case

testfolders.rev erse();
for t in testfolders:
do('mh.deletefo lder(%s)' % `t`)

The PEP suggests that

for t in testfolders.ite r_backwards():
do('mh.deletefo lder(%s)' % `t`)

is more beautiful and clearer than

for t in testfolders[::-1]:
do('mh.deletefo lder(%s)' % `t`)

(It definitely is more memory efficient if there are enough
folders.)

I don't think I agree with that. If I'm not concerned about memory
efficiency then extended slicing is nice because I don't have to
learn nor explain one more thing. ("There should be one ...")

The beauty and clarity, at least to me, only come into play
when memory is a concern, which is only rarely true for most
of the times iter_backwards is expected to be used. (Estimate
based on the use cases and my own judgement.)

] Should file objects be included? Implementing reverse iteration
] may not be easy though it would be useful on occasion.

I would suggest no - the file may not be seekable.

You suggest changing
] while _exithandlers:
] func, targs, kargs = _exithandlers.p op()
to
] for func, target, kargs in _exithandlers.i ter_backwards() :
] . . .
] del _exithandlers

One thing to bear in mind is that the semantics are slightly
different. In the first one, the finalizers for func, targs, kargs
*may* be called from last to first. This is the case for
CPython if there are no other references to the _exithandlers
objects. But since Python-the-language makes no guarantees
on that and Jython doesn't support it, it's a bad idea to write
code that way.

That is only a very, very minor nit. Feel free to ignore it :)


In difflib, I disagree that
result.sort()
return [x for score, x in result[-n:].iter_backwards ()]

is easier to understand than
result.sort() # Retain only the best n.
result = result[-n:] # Move best-scorer to head of list.
result.reverse( ) # Strip scores.
return [x for score, x in result]

The former is more terse, but the latter, existing code is
broken down into bite-size chunks and allows verbose
documentation of each step. That isn't possible when
nearly everything is written in one line.

Also, a variant implementation may be even more concise,
which is to replace
result.append(( s.ratio(), x))
with
result.append((-s.ratio(), x))
then get rid of the reverse (since the minus sign reverses the
sort order) and simply do

result.sort()
return [x for score, x in result[:n]]


] heapq.heapify() uses for i in xrange(n//2 - 1, -1, -1)

The proposed form is

for i in xrange(n//2-1).iter_backwar ds():

Later in your post you mention a reason against a function
which uses __riter__ magic is that it is "slower than direct
access to the underlying object."

If that's a concern then you should also point out that using
iter_backwards for cases like this is slower than the existing
solution. (It has an extra attr lookup and function call. OTOH,
doesn't the -1, -1 require two calls of the negative unary op
code, or has that been fixed? Good, it has, says dis on
some test code.)

] platform._dist_ try_harder() uses for n in range(len(verfi les)-1,-1,-1)
] because the loop deletes selected elements from verfiles but needs
] to leave the rest of the list intact for further iteration. This use case
] could be addressed with itertools.ifilt er() but would require the
] selection predicate to be in a lambda expression. The net result is
] less clear and readable than the original. A better reformulation is
] to replace the first line with the proposed method.

Are you sure about that? The code is

verfiles = os.listdir('/usr/lib/setup')
for n in range(len(verfi les)-1, -1, -1):
if verfiles[n][:14] != 'slack-version-':
del verfiles[n]

I think a better reformulation is

verfiles = os.listdir('/usr/lib/setup')
verfiles = [name for name in verfiles
if name.startswith ("slack-version-")]

(you could put the os.listdir in the list comp. but I've
found that gets ungainly.)

] random.shuffle( ) uses for i in xrange(len(x)-1, 0, -1)

This isn't a use case. The xrange never returns a 0 while
the iter_backwards one will.
[color=blue][color=green][color=darkred]
>>> list(xrange(5, 0, -1))[/color][/color][/color]
[5, 4, 3, 2, 1][color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

] Rejected Alternative Ideas

There were two intermediate 'reverse' proposals. Here's the four
I know about:

- call the magic method __riter__

- call the magic method __riter__. If it doesn't exist, use
for i in xrange(len(obj) ): yield obj[i]

- call the magic method __riter__. If it doesn't exist, use
for i in itertools.count (): yield[obj[-i]]

(this has the advantage of supporting the infinite sequence
0, 1, 2, ... -3, -2, -1
)

- same as the 2nd but if len doesn't exist use
data = list(obj)
data.reverse()
for x in data: yield x

I agree with the conclusion that the last three have unexpected
interactions with mappings. However, iter *also* has unexpected
interactions with mappings.
[color=blue][color=green][color=darkred]
>>> class Dict:[/color][/color][/color]
.... def keys(self):
.... return [0, 1, 2, 4]
.... def __getitem__(sel f, key):
.... if key not in self.keys(): raise KeyError(key)
.... return key * 2
.... def values(self):
.... return [self[x] for x in self.keys()]
.... def items(self): return zip(self.keys() , self.values())
.... def len(self): return len(self.keys() )
....[color=blue][color=green][color=darkred]
>>> for x in Dict():[/color][/color][/color]
.... print x
....
0
2
4
Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
File "<interacti ve input>", line 5, in __getitem__
KeyError: 3[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

While I think I was the one who originally raised this
objection, given the way Python already behaves wrt
mappings, I retract my objection.

Another possibility, which I just thought of now, is
- call the magic method __riter__. If it doesn't exist, look
for the reverse-iterator adapter, as per PEP-0246.

I don't have any experience with the PEP to judge its
appropriateness nor usefulness. The motivation section
seems to suggest that it would be appropriate.


In your post list a set of problems with the function approaches.
[color=blue]
> It saves implementing some object methods at the
> expense of adding a new builtin function and of
> creating a new magic method name.
> It is slower than direct access to the underlying object.
> It crashes when applied to an infinite iterator.
> It produces bizarre results when applied to mappings.[/color]

I would like to see this list added to the PEP. I do have'
some qualifiers for the list.

- it doesn't need to be a global function; it could be in,
say, itertools. (I know I originally suggested it as a builtin,
but I'm not saying I'm right ;)

- (see comments above that xrange(5).iter_ backwards()
is likely slower than xrange(5, -1, -1) because of the
extra lookup and call)

- the first variant, which only looks for __riter__ and raises
an exception if it doesn't exist, has exactly the same problems
with infinite iterators as calling the new method.

- similarly, it doesn't have additional problems with mappings.


Also, none of the use cases dealt with infinite sequences so
the lack of appropriate support for them shouldn't be considered
an outright rejection.

BTW, my current belief is that if this PEP leads to code
then it should be:
- a new function, either builtin or in itertools

- which uses __riter__ if present (unless we decide to
use the adapt PEP)

- else uses "for i in itertools.count (): yield[obj[-i]]"


Andrew
dalke@dalkescie ntific.com

[Andrew Dalke]

Re: Comment on PEP-0322: Reverse Iteration Methods

Me:[color=blue]
> - else uses "for i in itertools.count (): yield[obj[-i]]"[/color]

should be

- else uses "for i in itertools.count (1): yield[obj[-i]]"

Andrew
dalke@dalkescie ntific.com

[sebastien]

Re: Comment on PEP-0322: Reverse Iteration Methods

The PEP has the following sentence in the rejected alternatives:
[color=blue]
> * Add a builtin function, reverse() which calls a magic method, __riter__.
> I see this as more overhead for no additional benefit.[/color]

I believe that this is the better choice because it is the way forward
iteration work in python: you have the iter() function which call a
__iter__ special method.

So, I'd expect to have a riter() function which would call the
__riter__ special method.

I like riter as name to all other alternative because (iter, riter)
looks really like (find, rfind) or (index, rindex) ....

And I still think you don't need it often enough to put it in the
builtin namespace, so the function should go in the itertools module.

[Steve Holden]

Re: Comment on PEP-0322: Reverse Iteration Methods

"Raymond Hettinger" <vze4rx4y@veriz on.net> wrote ...[color=blue]
> Please comment on the new PEP for reverse iteration methods.[/color]

Looks good. But

[...][color=blue]
> It crashes when applied to an infinite iterator.[/color]
[...]
Hmm. My little brain is having difficulty imagining anything that won't.
What am I missing?

regards
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/pwp/

[Raymond Hettinger]

Re: Comment on PEP-0322: Reverse Iteration Methods

"sebastien" <s.keim@laposte .net> wrote in message[color=blue]
> So, I'd expect to have a riter() function which would call the
> __riter__ special method.[/color]
.. . .[color=blue]
> And I still think you don't need it often enough to put it in the
> builtin namespace, so the function should go in the itertools module.[/color]

If you make a magic method, then the function calling it
needs to be a builtin. They go hand in hand.


Raymond

[Raymond Hettinger]

Re: Comment on PEP-0322: Reverse Iteration Methods

[Andrew Dalke][color=blue]
> However, the example code will work on a string, because list(string)
> returns a list, which does have a reverse.[/color]

Right. I'll remove the comment.

[color=blue]
> ] Extending slicing minimizes the code overhead but does nothing
> ] for memory efficiency, beauty, or clarity.
> I don't agree with all three of those.[/color]

Beauty and clarity are a matter of taste and we differ here.
To some (including me), s[::-1] looks more like line noise
than s.iter_backward s(). Also, I think the word form is
more clear. The situation becomes more acute when the
other indices have values. Compare range[n-1, 0, -1] to
range(1, n-1).iterbackward s(). Whenever negative indicies
are used, it takes more knowledge of the implementation
rules to be able to read,write, understand, and verify the code.

[color=blue]
> ] Should file objects be included? Implementing reverse iteration
> ] may not be easy though it would be useful on occasion.
>
> I would suggest no - the file may not be seekable.[/color]

Agreed.


[color=blue]
> One thing to bear in mind is that the semantics are slightly
> different.[/color]
.. . .[color=blue]
> That is only a very, very minor nit. Feel free to ignore it :)[/color]

Agreed. If finalizer call order is important, then the
'while elist: elist.pop()' form is the way to go. If not,
then the for-loop is the way to go.

[color=blue]
> The former is more terse, but the latter, existing code is
> broken down into bite-size chunks and allows verbose
> documentation of each step. That isn't possible when
> nearly everything is written in one line.[/color]

Even the former can be broken down into more lines if
that is what is needed:

result.sort()
result = result[-n]
return [x for score, x in result.iter_bac kwards()]

[color=blue]
> Also, a variant implementation may be even more concise,[/color]

You must be pretty confident to take on the Timbot's code ;-)

[color=blue]
> Later in your post you mention a reason against a function
> which uses __riter__ magic is that it is "slower than direct
> access to the underlying object."
>
> If that's a concern then you should also point out that using
> iter_backwards for cases like this is slower than the existing
> solution. (It has an extra attr lookup and function call. OTOH,
> doesn't the -1, -1 require two calls of the negative unary op
> code, or has that been fixed? Good, it has, says dis on
> some test code.)[/color]

Yes, it's true. All of the speed comments focus on the inner loop.
I'll reword to make that clear. It is definitely an issue. In Py2.2,
there was a generic sequence iterator that used __getitem__ in
iterators for lists and tuples. In Py2.3, we went to a custom
iterator for each object. There was a significant improvement
in performance.


[color=blue]
> ] platform._dist_ try_harder()[/color]
. . .[color=blue]
> I think a better reformulation is
>
> verfiles = os.listdir('/usr/lib/setup')
> verfiles = [name for name in verfiles
> if name.startswith ("slack-version-")][/color]

I like your version better and recommend you submit it as a patch.
I'll take out the ifilter() comment.


[color=blue]
> ] random.shuffle( ) uses for i in xrange(len(x)-1, 0, -1)
> This isn't a use case. The xrange never returns a 0 while
> the iter_backwards one will.[/color]

It is an important use case. The replacement code is:

for i in xrange(1,len(x) ). iter_backwards( )

Whenever the indices have any complexity, the forwards version,
followed by .iter_backwards () is, IMHO, much easier to mentally verify.

[color=blue]
> There were two intermediate 'reverse' proposals. Here's the four
> I know about:[/color]

I'll try to add these without making the PEP as long as this thread ;-)

[color=blue]
> I agree with the conclusion that the last three have unexpected
> interactions with mappings. However, iter *also* has unexpected
> interactions with mappings.[/color]

Yes, but that doesn't make it less of a disadvantage.
The proposed list.iter_backw ards() method does not have that
disadvantage.


[color=blue]
> In your post list a set of problems with the function approaches.[/color]
....[color=blue]
> I would like to see this list added to the PEP.[/color]

Okay. They are added.

[color=blue]
> I do have'
> some qualifiers for the list.
>
> - it doesn't need to be a global function; it could be in,
> say, itertools. (I know I originally suggested it as a builtin,
> but I'm not saying I'm right ;)[/color]

If there is to be a magic method, __riter__, then the access
function needs to be a builtin. They go hand in hand.

BTW, don't underestimate the intensity of resistance to adding
new builtins.

[color=blue]
> - (see comments above that xrange(5).iter_ backwards()
> is likely slower than xrange(5, -1, -1) because of the
> extra lookup and call)[/color]

Reworded it to talk only to the inner loop, "approaches using
__getitem__() are slower using a custom method for
each object."

[color=blue]
> - the first variant, which only looks for __riter__ and raises
> an exception if it doesn't exist, has exactly the same problems
> with infinite iterators as calling the new method.
>
> - similarly, it doesn't have additional problems with mappings.[/color]

Okay, I've moved the comments around for you.

[color=blue]
> Also, none of the use cases dealt with infinite sequences so
> the lack of appropriate support for them shouldn't be considered
> an outright rejection.[/color]

The use cases were just the ones I found in the library.
With generators and itertools being new, I expect
infinite iterators to become more common.

Also, I have a preference for creating something that
is as robust as possible. Making a builtin function that
doesn't apply to all objects; behaves badly with mappings;
and crashes with an infinite iterator is not my idea of robust.

I really don't understand the overpowering urge to make this a function
and add a new magic method protocol. Why isn't there a similar rage
against dict.iteritems( )?

Perhaps, you'll like it better if the method is named ireverse()
instead of iter_backwards( ).

[color=blue]
> BTW, my current belief is that if this PEP leads to code
> then it should be:
> - a new function, either builtin or in itertools
>
> - which uses __riter__ if present (unless we decide to
> use the adapt PEP)
>
> - else uses "for i in itertools.count (): yield[obj[-i]]"[/color]

Okay, we simply disagree here. That's fine.

I appreciate the thoughtful comments and careful analysis.
They have helped clarify the pep wording and helped to
define the issues.


Raymond

[Raymond Hettinger]

Re: Comment on PEP-0322: Reverse Iteration Methods

[Stephen Horne][color=blue]
> The question is - is the exercise worthwhile. I'm not sure even though
> it's my own idea. I think it is nice in its way, and a lot more
> flexible than the 'iter_backward' method proposal, but that is quite
> likely a bad thing as it is probably massive overkill for the problem
> being solved - and in particular the extra flexibility has costs.
>
> What the hell, though - it was fun to play with for a bit![/color]

Yes, it's a one ton solution to a one ounce problem,
but it was an interesting exercise and diversion.



Raymond Hettinger

[Raymond Hettinger]

Re: Comment on PEP-0322: Reverse Iteration Methods

[sebastien][color=blue]
> So, I'd expect to have a riter() function which would call the
> __riter__ special method.[/color]

Okay, changed pep to list riter() as an alternative.

[color=blue]
> And I still think you don't need it often enough to put it in the
> builtin namespace, so the function should go in the itertools module.[/color]

If you have a magic method, __riter__, then the corresponding
function needs to be a builtin. They go hand in hand. The
core parts of the language need to be usable without having
to know about itertools.


Raymond

[Tom Anderson]

Re: Comment on PEP-0322: Reverse Iteration Methods

On Thu, 25 Sep 2003, Steve Holden wrote:
[color=blue]
> "Raymond Hettinger" <vze4rx4y@veriz on.net> wrote ...[color=green]
> > Please comment on the new PEP for reverse iteration methods.[/color]
>
> Looks good. But
>
> [...][color=green]
> > It crashes when applied to an infinite iterator.[/color]
> [...]
> Hmm. My little brain is having difficulty imagining anything that won't.
> What am I missing?[/color]

ideally, riter(riter(som eInfiniteIterat or)) would work, though!

tom

--
I see large and small words on this page, arranged in long rows separated by little dotty characters. Suspect written by little dotty characters, too. -- RonJeffries

[Andrew Dalke]

Re: Comment on PEP-0322: Reverse Iteration Methods

Raymond Hettinger:[color=blue]
> Beauty and clarity are a matter of taste and we differ here.
> To some (including me), s[::-1] looks more like line noise
> than s.iter_backward s().[/color]

Understood. In all this, let me make it clearthat I
have a preference for a function over a method, but won't
call it a wart of the BFDL thinks otherwise. I point these
out because I'm rather detail oriented.
[color=blue][color=green]
> > Also, a variant implementation may be even more concise,[/color]
>
> You must be pretty confident to take on the Timbot's code ;-)[/color]

Well, I did say "may". I wouldn't really want to use it since
inverting the score before the sort is a tricky thing to see and
somewhat unexpected.

(I'm half expecting some enlightening comment about how
negation has subtle interactions with IEEE 754 math -- though
I thought the point of 754 was to make naive approaches like
mine usually correct. :)
[color=blue][color=green]
> > verfiles = os.listdir('/usr/lib/setup')
> > verfiles = [name for name in verfiles
> > if name.startswith ("slack-version-")][/color]
>
> I like your version better and recommend you submit it as a patch.
> I'll take out the ifilter() comment.[/color]

But that goes against the pretty good motto of "if it ain't broke,
don't touch it." Granted, XP-style tests are supposed to
help out with that, but I don't have a /usr/lib/setup where I
can test this.
[color=blue][color=green]
> > ] random.shuffle( ) uses for i in xrange(len(x)-1, 0, -1)
> > This isn't a use case. The xrange never returns a 0 while
> > the iter_backwards one will.[/color]
>
> It is an important use case. The replacement code is:
>
> for i in xrange(1,len(x) ). iter_backwards( )[/color]

Ahh, right. Didn't think about that. You should include that
wording in the PEP.
[color=blue]
> Whenever the indices have any complexity, the forwards version,
> followed by .iter_backwards () is, IMHO, much easier to mentally verify.[/color]

I agree. When I do need to go backwards I often forgot to get
both -1s in place. Ie, I'll do (n, 0) instead of (n-1, -1).
Well, don't forgot much and more, but there's some tension in
my mind as I worry if I got it correct or not.
[color=blue]
> BTW, don't underestimate the intensity of resistance to adding
> new builtins.[/color]

For me it's two things: remembering what all these new things
do, and attempting to adhere to the 'don't use variables with
the same name as builtins' guiding philosophy. I don't adhere
to the latter wrt the type named 'file'.
[color=blue]
> Also, I have a preference for creating something that
> is as robust as possible. Making a builtin function that
> doesn't apply to all objects; behaves badly with mappings;
> and crashes with an infinite iterator is not my idea of robust.[/color]

The backup which uses __getitem__ and negative offsets
does work for all objects, behaves exactly as badly as the
existing iter, and doesn't crash with an infinite iterator, no?

It's slow, but if you want the performance, make an __riter__.
[color=blue]
> I really don't understand the overpowering urge to make this a function
> and add a new magic method protocol. Why isn't there a similar rage
> against dict.iteritems( )?[/color]

Overpowering?

My preference is for a function because of:
- similaraties to iter, which is also a function
- there are many types of iteration orderings, and not all can be
special cased methods.
- I don't see backwards iteration as all that important

As for iteritems, I was annoyed about it because it meant my
dictionary-like code would become more complicated to
implement, until I came across UserDict.DictMi xin, which
lets user-defined code implement just a few methods then
builds the rest from there.

That suggests a UserList.ListMi xin might be useful, but it
isn't obvious to me that it would be.

I also note that items is a method so by similarity it makes
sense that iteritems also be a method. Additionally, many
things offer forward iteration, while only dicts offer items.

Should dicts support riteritems, ritervalues?
[color=blue]
> Okay, we simply disagree here. That's fine.[/color]

Yup. I don't think I have anything useful or even interesting
to offer to this dicusssion. I also think that with these changes,
the PEP is pretty complete and solid.

Andrew
dalke@dalkescie ntific.com

[Stephen Horne]

Re: Comment on PEP-0322: Reverse Iteration Methods

On Thu, 25 Sep 2003 00:58:45 GMT, "Raymond Hettinger"
<vze4rx4y@veriz on.net> wrote:
[color=blue]
>Please comment on the new PEP for reverse iteration methods.[/color]

One more thought.

Having an iter_backwards method on xrange seems wrong. This suggests
multiple functions/methods...

To generate backward ranges
range_backwards function
xrange_backward s funtion

for i in range_backwards (10) :
...

To iterate an object backwards
either method for appropriate objects, or
iter_backward() function and __iter_backward __ magic method
(I don't mind abbreviating the 'iter', but I think the 'backward'
needs to be kept very obvious and not reduced to an easily missed
'r' prefix).

for i in iter_backwards (seq) :
...

To enumate backwards etc
Add an enumerate_backw ard function which requires the input to
support __len__.

for i in enumerate_backw ards (seq) :
...

For backward slicing
Add a function which translates the slice to a backward equivalent
before applying it...

for i in slice_backwards (seq, start, stop, step) :
...


--
Steve Horne

steve at ninereeds dot fsnet dot co dot uk