Comment on PEP-0322: Reverse Iteration Methods

[Alex Martelli]

Re: Comment on PEP-0322: Reverse Iteration Methods

David Abrahams wrote:
[color=blue]
> Alex Martelli <aleax@aleax.it > writes:
>[color=green]
>> David Abrahams wrote:
>> ...[color=darkred]
>>> Also, the idea of denying tuples the ability to reverse iterate seems
>>> arbitrary and capricious.[/color]
>>
>> Sure, but so is denying them, e.g., non-mutating methods such as
>> .index() and .count().[/color]
>
> Not IMO. Immutability is a very useful trait.[/color]

Sure, very useful indeed -- and why does YO suggest that add such
NON-mutating methods would damage immutability in the LEAST...?

[color=blue][color=green]
>> At least we're _consistently_ arbitrary and capricious!-)[/color]
>
> Not in this case.[/color]

Some amplification would be welcome, because the above comment
on immutability's usefulness is totally obscure to me in context.


Alex

[Stephen Horne]

Re: Comment on PEP-0322: Reverse Iteration Methods

On Fri, 26 Sep 2003 14:51:05 -0400, David Abrahams
<dave@boost-consulting.com> wrote:
[color=blue]
>Alex Martelli <aleax@aleax.it > writes:
>[color=green]
>> David Abrahams wrote:
>> ...[color=darkred]
>>> Also, the idea of denying tuples the ability to reverse iterate seems
>>> arbitrary and capricious.[/color]
>>
>> Sure, but so is denying them, e.g., non-mutating methods such as
>> .index() and .count().[/color]
>
>Not IMO. Immutability is a very useful trait.[/color]

Yes - and perfectly consistent with having *NON*-mutating methods such
as .index() and .count() ;-)

I always assumed that these were considered inconsistent with normal
use of tuples (which certainly I rarely need to get 'index' or
'count'-like results from).

Actually, they even seem a little odd in list, to be honest. I'd have
functions, not necessarily even in __builtins__, which work on any
sequence. They just don't seem like everyday operations that should be
built into the object.


--
Steve Horne

steve at ninereeds dot fsnet dot co dot uk

[Stephen Horne]

Re: Comment on PEP-0322: Reverse Iteration Methods

On Fri, 26 Sep 2003 14:42:28 -0400, "Sean Ross"
<sross@connectm ail.carleton.ca > wrote:
[color=blue]
>
>"David Eppstein" <eppstein@ics.u ci.edu> wrote in message
>news:eppstei n-AF4213.10592426 092003@news.ser vice.uci.edu...[color=green]
>> Your proposed name has an implication for the semantics.[/color]
>
>Do the implied semantics of ireverse() differ from those implied by the
>names iter_backwards( ), reverse_view(), riter(), etc.?
>Because, to me, they' re just different words for the same idea.[/color]

So far as I can see, it is having the function rather than the method
that creates the issues - that originated with David Abrahams post (at
least tracing back through this thread - it had been suggested
before).

However, your reply was implicitly supporting the use of a function
rather than a method. If 'ireverse' is imported from 'itertools' then
it cannot be a method of the object being reversed.

I think some wires have been slightly crossed - David Eppstein raised
a good point, but should perhaps have aimed it at David Abrahams.


--
Steve Horne

steve at ninereeds dot fsnet dot co dot uk

[Chad Netzer]

Re: Comment on PEP-0322: Reverse Iteration Methods

On Fri, 2003-09-26 at 11:42, Sean Ross wrote:
[color=blue][color=green]
> > I was questioning the feasibility of the implied semantics.[/color]
>
> Fine. Take it up with the person who proposed the idea.[/color]

You proposed it (hear me out). You are missing David's point. In
addition to whatever semantics a general reverse iterator might have, he
is saying that the itertools iterators have an implied additional
semantic of not (necessarily) needing to fully expand an iterable in
memory to operate on it, as a general ireverse() would probably have to
do. So, having ireverse potentially use up VAST quantities of memory,
in order to work, doesn't quite fit into the itertools philosophy.

That is the point he is making, and it is a specific comment on your (I
believe; others have probably proposed it as well) suggestion of an
ireverse() in itertools.

Now, my response to David's point is that currently, cycle() may also
require enough extra storage to remember an entire iterated sequence, so
the itertools philosophy is not a hard rule. Still, ireverse() would
imply some real devilry under the covers...

--
Chad Netzer

[Sean Ross]

Re: Comment on PEP-0322: Reverse Iteration Methods

"Chad Netzer" <cnetzer@sonic. net> wrote in message
news:mailman.10 64604518.1923.p ython-list@python.org ...[color=blue]
> On Fri, 2003-09-26 at 11:42, Sean Ross wrote:
>[color=green][color=darkred]
> > > I was questioning the feasibility of the implied semantics.[/color]
> >
> > Fine. Take it up with the person who proposed the idea.[/color]
>
> You proposed it (hear me out). You are missing David's point. In
> addition to whatever semantics a general reverse iterator might have, he
> is saying that the itertools iterators have an implied additional
> semantic of not (necessarily) needing to fully expand an iterable in
> memory to operate on it, as a general ireverse() would probably have to
> do. So, having ireverse potentially use up VAST quantities of memory,
> in order to work, doesn't quite fit into the itertools philosophy.
>
> That is the point he is making, and it is a specific comment on your (I
> believe; others have probably proposed it as well) suggestion of an
> ireverse() in itertools.
>
> Now, my response to David's point is that currently, cycle() may also
> require enough extra storage to remember an entire iterated sequence, so
> the itertools philosophy is not a hard rule. Still, ireverse() would
> imply some real devilry under the covers...
>
> --
> Chad Netzer
>
>[/color]

Hi.

Thanks for clearing that up.

Yes, "others have ... proposed it as well". I was merely interested in the
appearance of the code, not the underlying implementation. I was not aware
that the name would imply more than the other suggestions simply because it
was housed in itertools. But then that's not really the point, I suppose:
The main point is the "implied additional semantic", regardless of the
function's location. Yes? OK. Fine. I was not concerned with semantics when
I made the suggestion, only sugar, so I couldn't see where David was coming
from. Sorry for the confusion.

Sean

[David Abrahams]

Re: Comment on PEP-0322: Reverse Iteration Methods

Alex Martelli <aleax@aleax.it > writes:
[color=blue]
> David Abrahams wrote:
>[color=green]
>> Alex Martelli <aleax@aleax.it > writes:
>>[color=darkred]
>>> David Abrahams wrote:
>>> ...
>>>> Also, the idea of denying tuples the ability to reverse iterate seems
>>>> arbitrary and capricious.
>>>
>>> Sure, but so is denying them, e.g., non-mutating methods such as
>>> .index() and .count().[/color]
>>
>> Not IMO. Immutability is a very useful trait.[/color]
>
> Sure, very useful indeed -- and why does YO suggest that add such
> NON-mutating methods would damage immutability in the LEAST...?[/color]

Sorry, I misread your reply. I didn't know those were missing.
[color=blue][color=green][color=darkred]
>>> At least we're _consistently_ arbitrary and capricious!-)[/color]
>>
>> Not in this case.[/color]
>
> Some amplification would be welcome, because the above comment
> on immutability's usefulness is totally obscure to me in context.[/color]

My mistake; you were right as usual.

--
Dave Abrahams
Boost Consulting

https://www.boost-consulting.com

[David Abrahams]

Re: Comment on PEP-0322: Reverse Iteration Methods

Stephen Horne <$$$$$$$$$$$$$$ $$$@$$$$$$$$$$$ $$$$$$$$$.co.uk > writes:
[color=blue]
> On Fri, 26 Sep 2003 14:51:05 -0400, David Abrahams
> <dave@boost-consulting.com> wrote:
>[color=green]
>>Alex Martelli <aleax@aleax.it > writes:
>>[color=darkred]
>>> David Abrahams wrote:
>>> ...
>>>> Also, the idea of denying tuples the ability to reverse iterate seems
>>>> arbitrary and capricious.
>>>
>>> Sure, but so is denying them, e.g., non-mutating methods such as
>>> .index() and .count().[/color]
>>
>>Not IMO. Immutability is a very useful trait.[/color]
>
> Yes - and perfectly consistent with having *NON*-mutating methods such
> as .index() and .count() ;-)[/color]

Right. Oops, I misread.
[color=blue]
> I always assumed that these were considered inconsistent with normal
> use of tuples (which certainly I rarely need to get 'index' or
> 'count'-like results from).[/color]

I don't see why not.
[color=blue]
> Actually, they even seem a little odd in list, to be honest. I'd have
> functions, not necessarily even in __builtins__, which work on any
> sequence. They just don't seem like everyday operations that should be
> built into the object.[/color]

It's probably just for optimization purposes.

--
Dave Abrahams
Boost Consulting

https://www.boost-consulting.com

[David Abrahams]

Re: Comment on PEP-0322: Reverse Iteration Methods

"Sean Ross" <sross@connectm ail.carleton.ca > writes:
[color=blue]
> "David Eppstein" <eppstein@ics.u ci.edu> wrote in message
> news:eppstein-AF4213.10592426 092003@news.ser vice.uci.edu...[color=green]
>> Your proposed name has an implication for the semantics.[/color]
>
> Do the implied semantics of ireverse() differ from those implied by the
> names iter_backwards( ), reverse_view(), riter(), etc.?
> Because, to me, they' re just different words for the same idea.[/color]

Well, the only other place we have an 'i' prefix on a function name,
AFAIK, it means "inplace". I don't mind riter(); it has a nice
resonance with rbegin() in C++.

--
Dave Abrahams
Boost Consulting

https://www.boost-consulting.com

[Raymond Hettinger]

Re: Comment on PEP-0322: Reverse Iteration Methods

[Raymond][color=blue][color=green]
> > It crashes when applied to an infinite iterator.[/color][/color]

[Steve Holden][color=blue]
> Hmm. My little brain is having difficulty imagining anything that won't.
> What am I missing?[/color]

The PEP proposes adding a method only to a few objects
that don't have degenerate cases: list, str, unicode, xrange
and possibly tuple.


Raymond Hettinger

[Raymond Hettinger]

Re: Comment on PEP-0322: Reverse Iteration Methods

[Andrew Dalke][color=blue][color=green][color=darkred]
> > > verfiles = os.listdir('/usr/lib/setup')
> > > verfiles = [name for name in verfiles
> > > if name.startswith ("slack-version-")][/color]
> >
> > I like your version better and recommend you submit it as a patch.
> > I'll take out the ifilter() comment.[/color]
>
> But that goes against the pretty good motto of "if it ain't broke,
> don't touch it."[/color]

Between major releases, refactoring is fair game. Using newer
constructs like list comprehensions can make this code more
readable and maintainable.

[color=blue]
> Granted, XP-style tests are supposed to
> help out with that, but I don't have a /usr/lib/setup where I
> can test this.[/color]

Submit a patch, assign to me, and I'll test it.

[color=blue][color=green][color=darkred]
> > > > random.shuffle( ) uses for i in xrange(len(x)-1, 0, -1)
> > > This isn't a use case. The xrange never returns a 0 while
> > > the iter_backwards one will.[/color]
> >
> > It is an important use case. The replacement code is:
> >
> > for i in xrange(1,len(x) ). iter_backwards( )[/color]
>
> Ahh, right. Didn't think about that. You should include that
> wording in the PEP.[/color]

Okay, done!

[color=blue][color=green]
> > Whenever the indices have any complexity, the forwards version,
> > followed by .iter_backwards () is, IMHO, much easier to mentally verify.[/color]
>
> I agree. When I do need to go backwards I often forgot to get
> both -1s in place. Ie, I'll do (n, 0) instead of (n-1, -1).
> Well, don't forgot much and more, but there's some tension in
> my mind as I worry if I got it correct or not.[/color]

That is a key motivation for the pep!

[color=blue][color=green]
> > Also, I have a preference for creating something that
> > is as robust as possible. Making a builtin function that
> > doesn't apply to all objects; behaves badly with mappings;
> > and crashes with an infinite iterator is not my idea of robust.[/color]
>
> The backup which uses __getitem__ and negative offsets
> does work for all objects, behaves exactly as badly as the
> existing iter, and doesn't crash with an infinite iterator, no?[/color]

1. Sure, all objects that define __getitem__ in a non-mapping
sense or that don't ignore the index (that was at one time a
common python practice).

2. With infinite iterators, running them to completion causes
the crash. They can be used in other generators, iterators,
or loops that have a break. This is not true with reverse
iterators where the very first call will cause the crash:

itertools.count ().next() # this always works
riter(itertools .count()).next( ) # this always fails

[color=blue]
> That suggests a UserList.ListMi xin might be useful, but it
> isn't obvious to me that it would be.[/color]

You're the fourth person to suggest it.
If you want to know the issues involved, see my post at:

http://groups.google.com/groups?selm=Bebab.9323%24hl4.4031@nwrdny01.gnilink.net

[color=blue]
> Should dicts support riteritems, ritervalues?[/color]

Heck no! I only want reverse iteration for lists, strings, and xrange
where forward and reverse orders make some sense.

[color=blue]
> I also think that with these changes,
> the PEP is pretty complete and solid.[/color]

Thanks for the detailed help getting it to that point.


Raymond

[David Eppstein]

Re: Comment on PEP-0322: Reverse Iteration Methods

In article <dp6db.7740$541 .7674@nwrdny02. gnilink.net>,
"Raymond Hettinger" <vze4rx4y@veriz on.net> wrote:
[color=blue]
> [Raymond][color=green][color=darkred]
> > > It crashes when applied to an infinite iterator.[/color][/color]
>
> [Steve Holden][color=green]
> > Hmm. My little brain is having difficulty imagining anything that won't.
> > What am I missing?[/color]
>
> The PEP proposes adding a method only to a few objects
> that don't have degenerate cases: list, str, unicode, xrange
> and possibly tuple.[/color]

Speaking of which: str, int, etc., used to be functions, but they got
changed to be the initializers of types having those names. Why is
xrange still a function rather than the initializer of the xrange type?

The most visible difference of changing xrange to be the type of the
xrange objects would be more information shown in help(xrange), some of
which could be useful if we are talking about adding more methods to the
xrange objects.

--
David Eppstein http://www.ics.uci.edu/~eppstein/
Univ. of California, Irvine, School of Information & Computer Science

[Stephen Horne]

Re: Comment on PEP-0322: Reverse Iteration Methods

On Sat, 27 Sep 2003 00:08:22 -0700, David Eppstein
<eppstein@ics.u ci.edu> wrote:
[color=blue]
>Speaking of which: str, int, etc., used to be functions, but they got
>changed to be the initializers of types having those names. Why is
>xrange still a function rather than the initializer of the xrange type?[/color]

"""
Python 2.3 (#46, Jul 29 2003, 18:54:32) [MSC v.1200 32 bit (Intel)] on
win32
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> xrange[/color][/color][/color]
<type 'xrange'>
"""

Looks like a type rather than a function to me, and the help appears
to be very good. Enumerate also seems to be a type with an
initialiser.

Are you using an earlier version?


--
Steve Horne

steve at ninereeds dot fsnet dot co dot uk

[Alex Martelli]

Re: Comment on PEP-0322: Reverse Iteration Methods

David Abrahams wrote:
...[color=blue][color=green]
>> I always assumed that these were considered inconsistent with normal
>> use of tuples (which certainly I rarely need to get 'index' or
>> 'count'-like results from).[/color]
>
> I don't see why not.[/color]

The party line (not saying I necessarily agree, mind you -- just relating
it) is that tuples are meant as semantically heterogeneous containers
(semantically in that the items may happen to be the same type, but
their _meaning_ is disparate: e.g., consider the tuples used in modules
time or stat). So it doesn't really make sense to iterate on them, either;
it just happens to be possible because their items are accessed with []
with progressive naturals, and they have a len(). Associating _names_
to the indices makes more sense and indeed such pseudo-tuples are
starting to be supplied by the standard library (though not a general
mechanism for making your own, yet). Much like you can't iterate in
C++ on a std::pair<>, thus you "shouldn't" be able to iterate on a tuple.

"Frozen" versions of lists and dicts might make more sense -- and lose
only the MUTATING methods. I've seen Ruby experts claim that the
idea of freezing an object looks cool but in practice you end up almost
never using it -- they're speaking from experience, I guess, since in Ruby
they've had the possibility of freezing any object for ages. However, my
intuition (not supported by that much specific experience) makes me
yearn for such a feature...


Alex

[David Abrahams]

Re: Comment on PEP-0322: Reverse Iteration Methods

Alex Martelli <aleax@aleax.it > writes:
[color=blue]
> David Abrahams wrote:
> ...[color=green][color=darkred]
>>> I always assumed that these were considered inconsistent with normal
>>> use of tuples (which certainly I rarely need to get 'index' or
>>> 'count'-like results from).[/color]
>>
>> I don't see why not.[/color]
>
> The party line (not saying I necessarily agree, mind you -- just relating
> it) is that tuples are meant as semantically heterogeneous containers
> (semantically in that the items may happen to be the same type, but
> their _meaning_ is disparate: e.g., consider the tuples used in modules
> time or stat). So it doesn't really make sense to iterate on them, either;
> it just happens to be possible because their items are accessed with []
> with progressive naturals, and they have a len(). Associating _names_
> to the indices makes more sense and indeed such pseudo-tuples are
> starting to be supplied by the standard library (though not a general
> mechanism for making your own, yet). Much like you can't iterate in
> C++ on a std::pair<>, thus you "shouldn't" be able to iterate on a
> tuple.[/color]

Well, (understanding that you don't nececessarily agree with the
above) you can in fact iterate on std::pair<T,T> with the usual C++
iterator protocol, and with a new mixed compile-time/runtime tuple
iterator protocol developed by Doug Gregor, iteration over
heterogeneous tuples is possible too. It certainly is desirable to be
able to do that; it's a real need that has come up in practice.

--
Dave Abrahams
Boost Consulting

https://www.boost-consulting.com

[Alex Martelli]

Re: Comment on PEP-0322: Reverse Iteration Methods

David Abrahams wrote:
...[color=blue]
> Well, (understanding that you don't nececessarily agree with the
> above) you can in fact iterate on std::pair<T,T> with the usual C++
> iterator protocol,[/color]

You mean there's an std::pair::begi n etc?! OK, I guess I'm even
rustier on standard C++ than I thought I was -- I could have SWORN
there wasn't. (Std chapter & verse pls? I plan to win bets based
on this tidbit...!-). So I guess the lack of those in gcc is a
breach of the Standard on gcc's part...?
[color=blue]
> and with a new mixed compile-time/runtime tuple
> iterator protocol developed by Doug Gregor, iteration over
> heterogeneous tuples is possible too. It certainly is desirable to be
> able to do that; it's a real need that has come up in practice.[/color]

So, given an arbitrary struct, with fields (generally) of different types,
you can iterate field by field? That's basically what the "heterogene ous
tuple" is supposed to be equivalent to, in the "party line" I was relating
(although the names to go with the fields are only present in a FEW such
tuples, such as those returned by modules time and stat, as of now; more
general tuples still haven't grown the ability to access fields by name,
in Python).

My partial dissent comes from feeling the need for "frozen/immutable
lists" and the fact that tuples are often used as such (including goofy
immutable representations of dicts, e.g. via tuple(thedict.i teritems()),
and the like). If tuples cannot be thought of as immutable lists, then
(I think) we need "immutable/hashable/frozen" lists by other means.
(I need to say "I think" because, as I quoted, Ruby experts claim that
the "anyobject.free ze" feature they do have isn't actually as useful
as it SEEMS it should be -- though I'm not sure I understand why, yet).


Alex