removing list comprehensions in Python 3.0

[Peter Hansen]

Re: removing list comprehensions in Python 3.0

Bengt Richter wrote:[color=blue]
> On Fri, 08 Jul 2005 22:29:30 -0600, Steven Bethard <steven.bethard @gmail.com> wrote:[color=green]
>>(1) There's no reason to get uncomfortable even if they're removed.
>>You'd just replace [] with list().[/color]
>
> So list(1, 2, 3) will be the same as [1, 2, 3] ??[/color]

No, the discussion is about list comprehensions. [1,2,3] is not a list
comprehension, as you know.

-Peter

[Raymond Hettinger]

Re: removing list comprehensions in Python 3.0

In all probability, both list comprehensions and generator expressions
will be around in perpetuity. List comps have been a very successful
language feature.

The root of this discussion has been the observation that a list
comprehension can be expressed in terms of list() and a generator
expression. However, the former is faster when you actually want a
list result and many people (including Guido) like the square brackets.

After the advent of generators, it seemed for a while that all
functions and methods that returned lists would eventually return
iterators instead. What we are learning is that there is a place for
both. It is darned inconvenient to get an iterator when you really
need a list, when you want to slice the result, when you want to see a
few elements through repr(), and when you need to loop over the
contents more than once.


Raymond Hettinger

[Bengt Richter]

Re: removing list comprehensions in Python 3.0

On Sat, 09 Jul 2005 10:16:17 -0400, Peter Hansen <peter@engcorp. com> wrote:
[color=blue]
>Bengt Richter wrote:[color=green]
>> On Fri, 08 Jul 2005 22:29:30 -0600, Steven Bethard <steven.bethard @gmail.com> wrote:[color=darkred]
>>>(1) There's no reason to get uncomfortable even if they're removed.
>>>You'd just replace [] with list().[/color]
>>
>> So list(1, 2, 3) will be the same as [1, 2, 3] ??[/color]
>
>No, the discussion is about list comprehensions. [1,2,3] is not a list
>comprehensio n, as you know.
>[/color]
D'oh. Sorry to have come in from contextual outer space ;-/

Regards,
Bengt Richter

[John Roth]

Re: removing list comprehensions in Python 3.0

"Raymond Hettinger" <python@rcn.com > wrote in message
news:1120919382 .609886.62680@g 14g2000cwa.goog legroups.com...[color=blue]
> In all probability, both list comprehensions and generator expressions
> will be around in perpetuity. List comps have been a very successful
> language feature.
>
> The root of this discussion has been the observation that a list
> comprehension can be expressed in terms of list() and a generator
> expression. However, the former is faster when you actually want a
> list result and many people (including Guido) like the square brackets.
>
> After the advent of generators, it seemed for a while that all
> functions and methods that returned lists would eventually return
> iterators instead. What we are learning is that there is a place for
> both. It is darned inconvenient to get an iterator when you really
> need a list, when you want to slice the result, when you want to see a
> few elements through repr(), and when you need to loop over the
> contents more than once.[/color]

I was wondering about what seemed like an ill-concieved rush to
make everything an iterator. Iterators are, of course, useful but there
are times when you really did want a list.

John Roth[color=blue]
>
>
> Raymond Hettinger
>[/color]

[George Sakkis]

Re: removing list comprehensions in Python 3.0

"Raymond Hettinger" <python@rcn.com > wrote:
[color=blue]
> In all probability, both list comprehensions and generator expressions
> will be around in perpetuity. List comps have been a very successful
> language feature.
>
> The root of this discussion has been the observation that a list
> comprehension can be expressed in terms of list() and a generator
> expression.[/color]

No, the root of the discussion, in this thread at least, was the answer
to "why not dict comprehensions ?", which was along the lines of "well,
you can do it in one line by dict(gen_expres sion)".
[color=blue]
> However, the former is faster when you actually want a
> list result and many people (including Guido) like the square brackets.[/color]

Also dict comprehensions are faster if you actually want a dict result,
set comprehensions for set result, and so on and so forth.
[color=blue]
> After the advent of generators, it seemed for a while that all
> functions and methods that returned lists would eventually return
> iterators instead. What we are learning is that there is a place for
> both.[/color]

Altering the result type of existing functions and methods is not
exactly the same with the discussion on the future of list
comprehensions; the latter affects only whether listcomps are special
enough to be granted special syntax support, when there is an
equivalent way to express the same thing. It's funny how one of the
arguments for removing lambda -- you can do the same by defining a
named function -- does not apply for list comprehensions.
[color=blue]
> It is darned inconvenient to get an iterator when you really
> need a list, when you want to slice the result, when you want to see a
> few elements through repr(), and when you need to loop over the
> contents more than once.
>
> Raymond Hettinger[/color]

Similar arguments can be given for dict comprehensions as well.

George

[John Roth]

Re: removing list comprehensions in Python 3.0

"George Sakkis" <gsakkis@rutger s.edu> wrote in message
news:1120927567 .463458.289770@ g49g2000cwa.goo glegroups.com.. .
[color=blue]
> It's funny how one of the
> arguments for removing lambda -- you can do the same by defining a
> named function -- does not apply for list comprehensions.[/color]

Which is a point a number of people have made many times,
with about as much effect as spitting into the wind.

Making a piece of functionality less convenient simply to
satisfy someone's sense of language esthetics doesn't seem
to me, at least, to be a really good idea.

John Roth

[color=blue]
>
>
> George
>[/color]

[Steven Bethard]

Re: removing list comprehensions in Python 3.0

Devan L wrote:[color=blue][color=green][color=darkred]
>>>>import timeit
>>>>t1 = timeit.Timer('l ist(i for i in xrange(10))')
>>>>t1.timeit ()[/color][/color]
>
> 27.267753024476 576
>[color=green][color=darkred]
>>>>t2 = timeit.Timer('[i for i in xrange(10)]')
>>>>t2.timeit ()[/color][/color]
>
> 15.050426800054 197
>[color=green][color=darkred]
>>>>t3 = timeit.Timer('l ist(i for i in xrange(100))')
>>>>t3.timeit ()[/color][/color]
>
> 117.61078097914 682
>[color=green][color=darkred]
>>>>t4 = timeit.Timer('[i for i in xrange(100)]')
>>>>t4.timeit ()[/color][/color]
>
> 83.502424470149 151
>
> Hrm, okay, so generators are generally faster for iteration, but not
> for making lists(for small sequences), so list comprehensions stay.[/color]

Ahh, thanks. Although, it seems like a list isn't very useful if you
never iterate over it. ;)

Also worth noting that in Python 3.0 it is quite likely that list
comprehensions and generator expressions will have the same underlying
implementation. So while your tests above satisfy my curiosity
(thanks!) they're not really an argument for retaining list
comprehensions in Python 3.0. And list comprehensions won't go away
before then because removing them will break loads of existing code.

STeVe

[Steven Bethard]

Re: removing list comprehensions in Python 3.0

Raymond Hettinger wrote:[color=blue]
> The root of this discussion has been the observation that a list
> comprehension can be expressed in terms of list() and a generator
> expression.[/color]

As George Sakkis already noted, the root of the discussion was actually
the rejection of the dict comprehensions PEP.
[color=blue]
> However, the former is faster when you actually want a list result[/color]

I would hope that in Python 3.0 list comprehensions and generator
expressions would be able to share a large amount of implementation, and
thus that the speed differences would be much smaller. But maybe not...
[color=blue]
> and many people (including Guido) like the square brackets.[/color]
^
|
This --------------------------+ of course, is always a valid point. ;)

STeVe

[Raymond Hettinger]

Re: removing list comprehensions in Python 3.0

[Steven Bethard][color=blue]
> I would hope that in Python 3.0 list comprehensions and generator
> expressions would be able to share a large amount of implementation, and
> thus that the speed differences would be much smaller. But maybe not...[/color]

Looking under the hood, you would see that the implementations are
necessarily as different as night and day. Only the API is similar.


Raymond

[Raymond Hettinger]

Re: removing list comprehensions in Python 3.0

[Raymond Hettinger][color=blue][color=green]
> > It is darned inconvenient to get an iterator when you really
> > need a list, when you want to slice the result, when you want to see a
> > few elements through repr(), and when you need to loop over the
> > contents more than once.[/color][/color]

[George Sakkis][color=blue]
> Similar arguments can be given for dict comprehensions as well.[/color]

You'll find that "lever" arguments carry little weight in Python
language design (well, you did X in place Y so now you have to do it
everywhere even if place Z lacks compelling use cases).

For each variant, the balance is different. Yes, of course, list
comprehensions have pros and cons similar to set comprehensions, dict
comps, etc. However, there are marked differences in frequency of use
cases, desirability of having an expanded form, implementation issues,
varying degrees of convenience, etc.

The utility and generality of genexps raises the bar quite high for
these other forms. They would need to be darned frequent and have a
superb performance advantage.

Take it from the set() and deque() guy, we need set, dict, and deque
comprehensions like we need a hole in the head. The constructor with a
genexp does the trick just fine.

Why the balance tips the other way for list comps is both subjective
and subtle. I don't expect to convince you by a newsgroup post.
Rather, I can communicate how one of the core developers perceives the
issue. IMHO, the current design strikes an optimal balance.

'nuff said,


Raymond

[Steven Bethard]

Re: removing list comprehensions in Python 3.0

Raymond Hettinger wrote:[color=blue]
> [Steven Bethard]
>[color=green]
>>I would hope that in Python 3.0 list comprehensions and generator
>>expressions would be able to share a large amount of implementation, and
>>thus that the speed differences would be much smaller. But maybe not...[/color]
>
> Looking under the hood, you would see that the implementations are
> necessarily as different as night and day. Only the API is similar.[/color]

Necessarily? It seems like list comprehensions *could* be implemented
as a generator expression passed to the list constructor. They're not
now, and at the moment, changing them to work this way seems like a bad
idea because list comprehensions would take a performance hit. But I
don't understand why the implementations are *necessarily* different.
Could you explain?

STeVe

P.S. The dis.dis output for list comprehensions makes what they're doing
pretty clear. But dis.dis doesn't seem to give me as much information
when looking at a generator expression:

py> def ge(items):
.... return (item for item in items if item)
....
py> dis.dis(ge)
2 0 LOAD_CONST 1 (<code object <generator
expression> at 0116FD20, file "<interacti ve input>", line 2>)
3 MAKE_FUNCTION 0
6 LOAD_FAST 0 (items)
9 GET_ITER
10 CALL_FUNCTION 1
13 RETURN_VALUE

I tried to grep through the dist\src directories for what a generator
expression code object looks like, but without any luck. Any chance you
could point me in the right direction?

[Bengt Richter]

Re: removing list comprehensions in Python 3.0

On Sat, 09 Jul 2005 22:32:22 -0600, Steven Bethard <steven.bethard @gmail.com> wrote:
[color=blue]
>Raymond Hettinger wrote:[color=green]
>> [Steven Bethard]
>>[color=darkred]
>>>I would hope that in Python 3.0 list comprehensions and generator
>>>expression s would be able to share a large amount of implementation, and
>>>thus that the speed differences would be much smaller. But maybe not...[/color]
>>
>> Looking under the hood, you would see that the implementations are
>> necessarily as different as night and day. Only the API is similar.[/color]
>
>Necessarily? It seems like list comprehensions *could* be implemented
>as a generator expression passed to the list constructor. They're not
>now, and at the moment, changing them to work this way seems like a bad
>idea because list comprehensions would take a performance hit. But I
>don't understand why the implementations are *necessarily* different.
>Could you explain?
>
>STeVe
>
>P.S. The dis.dis output for list comprehensions makes what they're doing
>pretty clear. But dis.dis doesn't seem to give me as much information
>when looking at a generator expression:
>
>py> def ge(items):
>... return (item for item in items if item)
>...
>py> dis.dis(ge)
> 2 0 LOAD_CONST 1 (<code object <generator
>expression> at 0116FD20, file "<interacti ve input>", line 2>)
> 3 MAKE_FUNCTION 0
> 6 LOAD_FAST 0 (items)
> 9 GET_ITER
> 10 CALL_FUNCTION 1
> 13 RETURN_VALUE
>
>I tried to grep through the dist\src directories for what a generator
>expression code object looks like, but without any luck. Any chance you
>could point me in the right direction?[/color]
[color=blue][color=green][color=darkred]
>>> import dis
>>> g = ge([1,2,0,3,'',4])
>>> dis.dis(g)[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "d:\python-2.4b1\lib\dis.p y", line 46, in dis
raise TypeError, \
TypeError: don't know how to disassemble generator objects

but:
[color=blue][color=green][color=darkred]
>>> dis.dis(ge)[/color][/color][/color]
2 0 LOAD_CONST 1 (<code object <generator expression> at 02EE4FA0, file "<stdin>", line 2>)
3 MAKE_FUNCTION 0
6 LOAD_FAST 0 (items)
9 GET_ITER
10 CALL_FUNCTION 1
13 RETURN_VALUE[color=blue][color=green][color=darkred]
>>> ge.func_code[/color][/color][/color]
<code object ge at 02EE4E60, file "<stdin>", line 1>[color=blue][color=green][color=darkred]
>>> ge.func_code.co _consts[/color][/color][/color]
(None, <code object <generator expression> at 02EE4FA0, file "<stdin>", line 2>)[color=blue][color=green][color=darkred]
>>> ge.func_code.co _consts[1][/color][/color][/color]
<code object <generator expression> at 02EE4FA0, file "<stdin>", line 2>[color=blue][color=green][color=darkred]
>>> dis.dis(ge.func _code.co_consts[1])[/color][/color][/color]
2 0 SETUP_LOOP 28 (to 31)
3 LOAD_FAST 0 ([outmost-iterable])[color=blue][color=green]
>> 6 FOR_ITER 21 (to 30)[/color][/color]
9 STORE_FAST 1 (item)
12 LOAD_FAST 1 (item)
15 JUMP_IF_FALSE 8 (to 26)
18 POP_TOP
19 LOAD_FAST 1 (item)
22 YIELD_VALUE
23 JUMP_ABSOLUTE 6[color=blue][color=green]
>> 26 POP_TOP[/color][/color]
27 JUMP_ABSOLUTE 6[color=blue][color=green]
>> 30 POP_BLOCK
>> 31 LOAD_CONST 0 (None)[/color][/color]
34 RETURN_VALUE

A little more info, anyway. HTH.

Regards,
Bengt Richter

[EP]

Re: removing list comprehensions in Python 3.0

[color=blue]
> Well, I want to offer a more radical proposal: why not free squared
> braces from the burden of representing lists at all? It should be
> sufficient to write
> [color=green][color=darkred]
> >>> list()[/color][/color]
> list()
> [/color]

<snip>
[color=blue]
>From a visual comprehenison point of view, I would assert that the square form [] is much easier on the eyes than the subtler curved forms (e.g. "{" and "(").[/color]

Burdened with old eyes, small fonts, and an old, inflexible mind (;-), one of Python features near and dear to me are lists, [], and list comprehensions, but perhaps a more important point for 3.0 would be that there is a seamless consistency across the language (e.g.[list(), dict(), tuple()] or [[], {}, () ] rather than[list(), {}, ()]) thus reflecting a cohesiveness both in underlying approach and symbology.

Even old guys can adjust to something new that is good and clean.

[Steven Bethard]

Re: removing list comprehensions in Python 3.0

Steven Bethard wrote:[color=blue]
> py> def ge(items):
> ... return (item for item in items if item)
> ...[/color]

Bengt Richter wrote:[color=blue][color=green][color=darkred]
> >>> dis.dis(ge)[/color][/color]
> 2 0 LOAD_CONST 1 (<code object <generator expression> at 02EE4FA0, file "<stdin>", line 2>)
> 3 MAKE_FUNCTION 0
> 6 LOAD_FAST 0 (items)
> 9 GET_ITER
> 10 CALL_FUNCTION 1
> 13 RETURN_VALUE[/color]
[snip][color=blue][color=green][color=darkred]
> >>> dis.dis(ge.func _code.co_consts[1])[/color][/color]
> 2 0 SETUP_LOOP 28 (to 31)
> 3 LOAD_FAST 0 ([outmost-iterable])[color=green][color=darkred]
> >> 6 FOR_ITER 21 (to 30)[/color][/color]
> 9 STORE_FAST 1 (item)
> 12 LOAD_FAST 1 (item)
> 15 JUMP_IF_FALSE 8 (to 26)
> 18 POP_TOP
> 19 LOAD_FAST 1 (item)
> 22 YIELD_VALUE
> 23 JUMP_ABSOLUTE 6[color=green][color=darkred]
> >> 26 POP_TOP[/color][/color]
> 27 JUMP_ABSOLUTE 6[color=green][color=darkred]
> >> 30 POP_BLOCK
> >> 31 LOAD_CONST 0 (None)[/color][/color]
> 34 RETURN_VALUE[/color]

Outstanding. Thanks a lot! For comparison, here's the relevant dis.dis
output for list comprehensions.

py> def lc(items):
.... return [item for item in items if item]
....
py> dis.dis(lc)
2 0 BUILD_LIST 0
3 DUP_TOP
4 STORE_FAST 1 (_[1])
7 LOAD_FAST 0 (items)
10 GET_ITER[color=blue][color=green]
>> 11 FOR_ITER 24 (to 38)[/color][/color]
14 STORE_FAST 2 (item)
17 LOAD_FAST 2 (item)
20 JUMP_IF_FALSE 11 (to 34)
23 POP_TOP
24 LOAD_FAST 1 (_[1])
27 LOAD_FAST 2 (item)
30 LIST_APPEND
31 JUMP_ABSOLUTE 11[color=blue][color=green]
>> 34 POP_TOP[/color][/color]
35 JUMP_ABSOLUTE 11[color=blue][color=green]
>> 38 DELETE_FAST 1 (_[1])[/color][/color]
41 RETURN_VALUE

Interestingly, the LC code and the code of a GE's "generator-expression"
code object look quite similar, with basically a LOAD_FAST/LIST_APPEND
replaced by a YIELD_VALUE.

But I don't know byte code well enough to guess how the dangling local
variable in LCs will be eliminated in Python 3.0 (as has been suggested
a number of times). One way to eliminate it would be (as suggested) to
make LCs syntactic sugar for list(<genexp>). But it also looks like it
might be possible to do a DELETE_FAST with an appropriately hidden name...

STeVe

[Edvard Majakari]

Re: removing list comprehensions in Python 3.0

Steven Bethard <steven.bethard @gmail.com> writes:
[color=blue]
> $ python -m timeit "for x in (i for i in xrange(10)): y = x"
> 100000 loops, best of 3: 4.75 usec per loop[/color]

Yowza! One of the features I really liked in Perl has shored Python island
somewhere in the 2.4'ies, it seems[1]. Thanks for the tip!

PS. In case it wasn't clear what I referred to, it was the ability to run
given module as a script. Of course you could supply full path to timeit.py:

$ python2.3 /usr/lib/python2.3/timeit.py \
"for x in [i for i in xrange(10)]: y = x"
100000 loops, best of 3: 9.96 usec per loop

But using -m makes it much more convenient.


Footnotes:
[1] Well, not exactly equal to -M in Perl, but close enough for timing stuff

--
# Edvard Majakari Software Engineer
# PGP PUBLIC KEY available Soli Deo Gloria!
You shouldn't verb verbs.