Generator expressions v/s list comprehensions

[Peter Hansen]

Re: Generator expressions v/s list comprehensions

A small correction:

Martin DeMello wrote:
[color=blue]
> Here's a simple example
>
> linecount = sum(1 for line in file)
> linecount = sum([1 for line in file])
>
> The former requires an in-memory list equal to the size of the file.[/color]
^^^^^^
I think you meant "the latter"...
[color=blue]
> Consigning "other than memory consumption" to a parenthetical remark
> misses its importance.[/color]

-Peter

[Martin DeMello]

Re: Generator expressions v/s list comprehensions

Peter Hansen <peter@engcorp. com> wrote:[color=blue]
> A small correction:
>
> Martin DeMello wrote:
>[color=green]
> > Here's a simple example
> >
> > linecount = sum(1 for line in file)
> > linecount = sum([1 for line in file])
> >
> > The former requires an in-memory list equal to the size of the file.[/color]
> ^^^^^^
> I think you meant "the latter"...[/color]

Oops - I did indeed.

martin

[Jeremy Bowers]

Re: Generator expressions v/s list comprehensions

On Mon, 30 Aug 2004 01:23:16 -0700, Paul Rubin wrote:
[color=blue]
> These things exist in other languages and had been
> requested in Python for years before they got accepted. And every
> time some little thing gets added changing the language, that creates
> a new mini-dialect that users have to remember for a while and then
> forget. The result is "version fatigue"; one gets bleary trying to
> remember what's in the language this week. Those features are in
> other languages for a reason, and there's been enough experience using
> them (in those languages) that their desirability for Python should
> never have seriously been in question.[/color]

Along with the other two poster's points, you have a whiff of an implied
argument that somehow, these amorphous "other languages" somehow *didn't*
go through "version fatigue", and are therefore somehow better. This is
false if you take it out of the realm of amorphous, implied claim and look
at it in the light of day. Show me a language that has never suffered from
these problems and anyone actually uses. If you can show me one, show me a
few more.

C changed. C++ changed. Fortran changed. COBOL changed. LISP has changed.
Python changed, Perl changed, Haskell changed, Javascript changed,
everything changed. None of those languages decided that the correct
solution is to simply throw everything in (even Ada, C++, and Perl, all of
which have that reputation, really didn't in the sense you're getting at
here).

The claim that other languages have features, *therefore* we shouldn't
stop to think about whether they are good and they should all be added to
Python without debate boggles the mind when viewed head on. Pure
non-sequitor.

(BTW, note levels of debate are proportional to how many people bother to
debate, and nothing else. It is one reason a language can not
effectively grow to the community size of Python without strong leadership
and why I am glad that Python runs under a BDFL system. Most of the debate
you may see here in c.l.p. is really effectively irrelevant, and should be
mentally categorized as such; that debate does not constitute true design
involvement. Thus, pointing out "high levels of debate" in the newsgroup
doesn't convince me of much, since it really means little. To the extent
that it does sometimes matter, the amortized effect of any given message
is simply miniscule; the entire debate of a thousand messages may have no
effect whatsoever on Guido, or manifest itself as a single document of
moderate length that he may *still* reject.)

[Michael J. Fromberger]

Re: Generator expressions v/s list comprehensions

In article <1gjbs1e.assxlx lud1m0N%aleaxit @yahoo.com>,
aleaxit@yahoo.c om (Alex Martelli) wrote:
[color=blue]
> Mahesh Padmanabhan <mahesh@privacy .net> wrote:[color=green]
> >
> > I just wish that generator expressions had been designed into the
> > language without going the way of list comprehension -> generator
> > expressions.[/color]
>
> It would sure be nice if Python had been born back from day one with all
> the neat features it has taken years to develop -- that way we wouldn't
> have any issues that are there just because of backwards compatibility.
>
> Unfortunately, this wish is totally unrealistic -- obviously people do
> come up with cool ideas such as the iterator protocol, and generators,
> after the language has been around for a while.[/color]

All the cool features Python has adopted existed a long time before they
made their way into Python itself. As I see it, the real issue with
taking up new language features is that the vast majority of work-a-day
programmers seem to be highly reticent to embrace new tools and ideas,
for one reason or another. I think this explains -- among other things
-- the lingering popularity of languages like BASIC (in its various
incarnations), and Fortran.

In fact, one of the things I love most about Python, both the language
and the community surrounding it, is that here, for the first time, we
see a large community of mainstream programmers who are ready, willing,
and able to adapt to new ideas without losing their heads and retreating
into the ostensibly safe and comfortable world of linear imperative
programming. Instead of insisting everybody forget what they know and
start over, however, Python gives us the old familiar stuff, too, and
the new ideas come in a little at a time.

Frankly, I think if the Lisp world had managed to build the same
friendly and welcoming community Python seems to have, it would have
taken over the world a quarter-century ago.

-M

--
Michael J. Fromberger | Lecturer, Dept. of Computer Science
http://www.dartmouth.edu/~sting/ | Dartmouth College, Hanover, NH, USA

[Alex Martelli]

Re: Generator expressions v/s list comprehensions

Michael J. Fromberger <Michael.J.From berger@Clothing .Dartmouth.EDU>
wrote:
...[color=blue][color=green]
> > Unfortunately, this wish is totally unrealistic -- obviously people do
> > come up with cool ideas such as the iterator protocol, and generators,
> > after the language has been around for a while.[/color]
>
> All the cool features Python has adopted existed a long time before they
> made their way into Python itself.[/color]

Almost all, not all. As I repeatedly explained, for example, generator
comprehensions are not quite like anything in other languages (though
you can surely find _similarities_, of course).

[color=blue]
> Frankly, I think if the Lisp world had managed to build the same
> friendly and welcoming community Python seems to have, it would have
> taken over the world a quarter-century ago.[/color]

I think most people's reaction to (==against) Lisp's parentheses and
prefix syntax is more intense than that to Python's significant
whitespace, which appears quite natural to most non-programmers even
though many programmers inured to block delimiters do react strongly
against it. It's hard enough for Python to keep its community welcoming
and friendly when you hear some clueless dweeb rant against indendation
for the thousandth time, I think the Lisp community would have had to be
made up of saints to be still friendly and welcoming after a _million_
newcomers raised their whining against parentheses.


Alex

[Steve Holden]

Re: Generator expressions v/s list comprehensions

Alex Martelli wrote:
[color=blue]
> Michael J. Fromberger <Michael.J.From berger@Clothing .Dartmouth.EDU>
> wrote:
> ...
>[color=green][color=darkred]
>>>Unfortunatel y, this wish is totally unrealistic -- obviously people do
>>>come up with cool ideas such as the iterator protocol, and generators,
>>>after the language has been around for a while.[/color]
>>
>>All the cool features Python has adopted existed a long time before they
>>made their way into Python itself.[/color]
>
>
> Almost all, not all. As I repeatedly explained, for example, generator
> comprehensions are not quite like anything in other languages (though
> you can surely find _similarities_, of course).
>[/color]
Indeed, and the reason I like Python is that it has some aspects of Icon
mixed in with perhaps the most easily-accessible object-oriented
features of any of the programming languages I've learned.[color=blue]
>
>[color=green]
>>Frankly, I think if the Lisp world had managed to build the same
>>friendly and welcoming community Python seems to have, it would have
>>taken over the world a quarter-century ago.[/color]
>
>
> I think most people's reaction to (==against) Lisp's parentheses and
> prefix syntax is more intense than that to Python's significant
> whitespace, which appears quite natural to most non-programmers even
> though many programmers inured to block delimiters do react strongly
> against it. It's hard enough for Python to keep its community welcoming
> and friendly when you hear some clueless dweeb rant against indendation
> for the thousandth time, I think the Lisp community would have had to be
> made up of saints to be still friendly and welcoming after a _million_
> newcomers raised their whining against parentheses.
>[/color]
.... which it clearly isn't :-).

I do think the Python community is just a little out-of-the-ordinary,
which is what keeps me (and, I suspect, you: great to see you on the
wires again) coming back to c.l.py.

regards
Steve

[Antoon Pardon]

Re: Generator expressions v/s list comprehensions

Op 2004-08-28, Alex Martelli schreef <aleaxit@yahoo. com>:[color=blue]
> Mahesh Padmanabhan <mahesh@privacy .net> wrote:
>[color=green]
>> Hi,
>>
>> When list comprehension was added to the language, I had a lot of
>> trouble understanding it but now that I am familiar with it, I am not
>> sure how I programmed in Python without it.[/color]
>
> Oh good.
>[color=green]
>>
>> Now I see that generator expressions have been added to the language
>> with 2.4 and I question the need for it. I know that it allows for lazy
>> evaluation which speeds things up for larger lists but why was it
>> necessary to add it instead of improving list comprehension?
>>
>> Was there some sort of limitation that prevented list comprehension from
>> taking over the functionality that generator expressions provide?[/color]
>
> Sure, and that limitation is: list comprehensions return lists. This
> one "limitation " (together with Python's applicative order evaluation,
> and you couldn't change THAT without breaking the whole caboodle of
> existing programs!) implies everything else.
>[color=green]
>>
>> I have always liked the fact that Python has limited capabilities for
>> having more than one way to do it and it seem to me that generator
>> expressions break that philosophy. It is similar to how you have to use
>> range or xrange depending on how large the range is.
>>
>> Can someone please explain the reasoning behind it?[/color]
>
> Generator comprehensions are wonderful and there is no way Python list
> comprehensions can provide the same features, since lists need to be
> lists. Sure, list(e(x) for x in foo) IS just the same thing as [e(x)
> for x in foo]. We'll remove the redundancy in 3.0 -- not earlier
> because it will break backwards compatibility. The only sensible way I
> can see right now for 3.0 to remove this redundancy is by removing list
> comprehensions and leaving only generator comprehensions, btw.[/color]

The one problem I have with generator comprehensions is that, because
list comprhension came first they look like tuple comprehensions.

In order to avoid this same kind of confusion with new comers I think
it would be best if generator comprehensions are explained first and
that list comprehension is just syntax sugar for list(genexp)

--
Antoon Pardon

[Raymond Hettinger]

Re: Generator expressions v/s list comprehensions

[Mahesh Padmanabhan][color=blue]
> Now I see that generator expressions have been added to the language
> with 2.4 and I question the need for it. I know that it allows for lazy
> evaluation which speeds things up for larger lists but why was it
> necessary to add it instead of improving list comprehension?[/color]

FWIW, we improved list comprehensions too. They run much faster now.


Raymond

[Nicolas Lehuen]

Re: Generator expressions v/s list comprehensions

python@rcn.com (Raymond Hettinger) wrote in message news:<5d83790c. 0408311434.3cc3 8c24@posting.go ogle.com>...[color=blue]
> [Mahesh Padmanabhan][color=green]
> > Now I see that generator expressions have been added to the language
> > with 2.4 and I question the need for it. I know that it allows for lazy
> > evaluation which speeds things up for larger lists but why was it
> > necessary to add it instead of improving list comprehension?[/color]
>
> FWIW, we improved list comprehensions too. They run much faster now.
>
>
> Raymond[/color]

IIRC, list constructors and list comprehensions now use the length of
the underlying sequence (if it defines __len__) to preallocate its
internal array.

Thus, the list comprehension is theoritically more efficient in this
case :

[x for x in range(100000)] vs [x for x in xrange(100000)]

So [x for x in whatever] may be preferable to list(x for x in
whatever) if you don't know what 'whatever' is and it happens in
define __len__.

BTW, I guess this is no longer true with an if statement :

[x for x in range(100000) if x%2==0] may be less efficient than [x for
x in xrange(100000) if x%2==0]

Regards,

Nicolas Lehuen

[Raymond Hettinger]

Re: Generator expressions v/s list comprehensions

> > FWIW, we improved list comprehensions too. They run much faster now.

[Nicolas Lehuen][color=blue]
> IIRC, list constructors and list comprehensions now use the length of
> the underlying sequence (if it defines __len__) to preallocate its
> internal array.[/color]

Yes for list().
No for list comps.


[color=blue]
> Thus, the list comprehension is theoritically more efficient[/color]

Aside from the theory being wrong, this is a testable assertion. Use
timeit.

The relative speed performance of genexps vs list comps depends on
length, cache, processor, memory allocator, use of re-usable
containers (tuples, dicts, etc), and other factors. Try this for
example:
dict( (i,i+i) for i in xrange(n) )

My recommendation is to always use list comps when you need a list and
to use genexps whenever values get consumed right away by an enclosing
function.


Raymond