generator object, next method

**Peter Otten** · Sep 8 '05, 07:55 AM

Re: generator object, next method

simonwittber@gm ail.com wrote:
[color=blue][color=green][color=darkred]
>>>> gen = iterator()
>>>> gen.next[/color][/color]
> <method-wrapper object at 0x009D1B70>[/color]

Behind the scene, gen.next is bound to _, i. e. it cannot be
garbage-collected. Then...
[color=blue][color=green][color=darkred]
>>>> gen.next[/color][/color]
> <method-wrapper object at 0x009D1BB0>[/color]

a new method wrapper is created and assigned to _, and the previous method
wrapper is now garbage-collected. The memory location is therefore
available for reuse for...
[color=blue][color=green][color=darkred]
>>>> gen.next[/color][/color]
> <method-wrapper object at 0x009D1B70>[/color]

yet another method wrapper -- and so on.
[color=blue][color=green][color=darkred]
>>>> gen.next[/color][/color]
> <method-wrapper object at 0x009D1BB0>[color=green][color=darkred]
>>>> gen.next is gen.next[/color][/color]
> False
>
>
> What is behind this apparently strange behaviour? (The .next method
> seems to alternately bind to two different objects)[/color]

But it isn't. What seems to be the same object are distinct objects at the
same memory location. See what happens if you inhibit garbage-collection by
keeping a reference of the method wrappers:
[color=blue][color=green][color=darkred]
>>> it = iter("")
>>> [it.next for _ in range(5)][/color][/color][/color]
[<method-wrapper object at 0x4029388c>, <method-wrapper object at
0x402938ec>, <method-wrapper object at 0x402938cc>, <method-wrapper object
at 0x4029390c>, <method-wrapper object at 0x4029392c>]

Peter

**Duncan Booth** · Sep 8 '05, 08:15 AM

Re: generator object, next method

simonwittber@gm ail.com wrote:
[color=blue]
>[color=green][color=darkred]
>>>> gen = iterator()
>>>> gen.next[/color][/color]
><method-wrapper object at 0x009D1B70>[color=green][color=darkred]
>>>> gen.next[/color][/color]
><method-wrapper object at 0x009D1BB0>[color=green][color=darkred]
>>>> gen.next[/color][/color]
><method-wrapper object at 0x009D1B70>[color=green][color=darkred]
>>>> gen.next[/color][/color]
><method-wrapper object at 0x009D1BB0>[color=green][color=darkred]
>>>> gen.next is gen.next[/color][/color]
> False
>
>
> What is behind this apparently strange behaviour? (The .next method
> seems to alternately bind to two different objects)[/color]

It is a combination of factors.

1) Every time you access gen.next you create a new method-wrapper object.
2) Typing an expression at the interactive prompt implicitly assigns the
result of the expression to the variable '_'
3) When the method-wrapper is destroyed the memory becomes available to be
reused the next time a method-wrapper (or other object of similar size) is
created.

So in fact you have 6 different objects produced by your 6 accesses to
gen.next although (since you never have more than 3 of them in existence at
a time) there are probably only 3 different memory locations involved.

**Paul Rubin** · Sep 8 '05, 08:25 AM

Re: generator object, next method

Duncan Booth <duncan.booth@i nvalid.invalid> writes:[color=blue]
> 1) Every time you access gen.next you create a new method-wrapper object.[/color]

Why is that? I thought gen.next is a callable and gen.next() actually
advances the iterator. Why shouldn't gen.next always be the same object?

**simonwittber@gmail.com** · Sep 8 '05, 08:25 AM

Re: generator object, next method

[color=blue]
> Why is that? I thought gen.next is a callable and gen.next() actually
> advances the iterator. Why shouldn't gen.next always be the same object?[/color]

That is, in essence, my question.

Executing the below script, rather than typing at a console, probably
clarifies things a little.

Sw.

#-------------------
def iterator():
yield None

gen = iterator()

#gen.next is bound to x, and therefore, gen.next should not be GC?
x = gen.next
y = gen.next
print x
print y
print gen.next
#-------------------

**Diez B. Roggisch** · Sep 8 '05, 08:45 AM

Re: generator object, next method

simonwittber@gm ail.com wrote:[color=blue][color=green]
>>Why is that? I thought gen.next is a callable and gen.next() actually
>>advances the iterator. Why shouldn't gen.next always be the same object?[/color]
>
>
> That is, in essence, my question.[/color]

Because bound methods are generated on the fly - google this group,
there have been plenty of discussions about that.

Diez

**Duncan Booth** · Sep 8 '05, 09:05 AM

Re: generator object, next method

Paul Rubin wrote:
[color=blue]
> Duncan Booth <duncan.booth@i nvalid.invalid> writes:[color=green]
>> 1) Every time you access gen.next you create a new method-wrapper
>> object.[/color]
>
> Why is that? I thought gen.next is a callable and gen.next() actually
> advances the iterator. Why shouldn't gen.next always be the same
> object?[/color]

It is a consequence of allowing methods to be first class objects, so
instead of just calling them you can also save the bound method in a
variable and call it with the 'self' context remembered in the method.

It is easier to see what is happening if you look first at ordinary Python
classes and instances:
[color=blue][color=green][color=darkred]
>>> class C:[/color][/color][/color]
def next(self): pass

[color=blue][color=green][color=darkred]
>>> c = C()
>>> c.next[/color][/color][/color]
<bound method C.next of <__main__.C instance at 0x00B4D6E8>>[color=blue][color=green][color=darkred]
>>> C.next[/color][/color][/color]
<unbound method C.next>

Here, 'next' is a method of the class C. You can call the unbound method,
but then you have to explicitly pass the 'self' argument.

Whenever you access the method through an instance it creates a new 'bound
method' object which stores references to both the original function, and
the value to be passed in as the first parameter. Usually this object is
simply called and discarded, but you can also save it for later use.

Python could perhaps bypass the creation of bound method objects when
calling a function directly, but it would still need them for cases where
the method isn't called immediately (and it isn't obvious it would be an
improvement if it tried to optimise this case).

It would be possible for a language such as Python to try to either
generate these bound method objects in advance (which would be horribly
inefficient if you created lots of objects each of which had hundreds of
methods which were never called), or to cache bound method objects so as to
reuse them (which would be inefficient if you have lots of methods called
only once on each object). Python chooses to accept the hit of creating
lots of small objects, but tries to make the overhead of this as low as
possible (which is one reason the memory gets reused immediately).

The next method in a generator works in the same way as bound methods,
although the actual types involved are C coded. You can still access both
the bound and unbound forms of the next method. The bound form carries the
information about the first parameter, and the unbound form has to be given
that information:
[color=blue][color=green][color=darkred]
>>> gen = iterator()
>>> gen.next[/color][/color][/color]
<method-wrapper object at 0x00B43E30>[color=blue][color=green][color=darkred]
>>> type(gen).next[/color][/color][/color]
<slot wrapper 'next' of 'generator' objects>[color=blue][color=green][color=darkred]
>>> gen.next()[/color][/color][/color]
'hi'[color=blue][color=green][color=darkred]
>>> type(gen).next( gen)[/color][/color][/color]
'hi'

**Terry Reedy** · Sep 8 '05, 06:15 PM

Re: generator object, next method

"Paul Rubin" <"http://phr.cx"@NOSPAM. invalid> wrote in message
news:7xll282cz4 .fsf@ruckus.bro uhaha.com...[color=blue]
> Duncan Booth <duncan.booth@i nvalid.invalid> writes:[color=green]
>> 1) Every time you access gen.next you create a new method-wrapper
>> object.[/color]
>
> Why is that? I thought gen.next is a callable and gen.next() actually
> advances the iterator. Why shouldn't gen.next always be the same object?[/color]

If you explicitly or implicitly (via for loop) calculate gen.next exact
once (as I presume for loops do, and as I would for explicit while loop),
then it is. When you keep a reference to the wrapper, and call it
repeatedly via that wrapper, then all is as you expect.

next_x = genfunc(*args). next
while True:
x = next_x() # same next_x each time
<do something with x>

Terry J. Reedy

generator object, next method

generator object, next method

Comment

Comment

Comment

Comment

Comment

Comment

Comment