My Generator Paradox!

[vbgunz]

I am afraid that this is the first time in which I would probably need
something explained to me as if I were a little child. I am having a
hard time getting this through my thick skull. What in the world is
wrong with this!?

''' ############### ############### ############### ############## '''

def generatorFuncti on(sequence=['item1', 'item2', 'item3']):
for item in sequence:
yield item

yieldedValue = generatorFuncti on()

'''this seems to work perfectly.'''
print '-' * 32
print yieldedValue # <generator object at 0xb723014c>
print yieldedValue.ne xt() # item1
print yieldedValue.ne xt() # item2
print yieldedValue.ne xt() # item3

'''this is where things don't make any sense!'''
print '-' * 32
print generatorFuncti on() # <generator object at 0xb723022c>
print generatorFuncti on().next() # item1
print generatorFuncti on().next() # item1
print generatorFuncti on().next() # item1

''' ############### ############### ############### ############## '''

the first set of calls assigned to yieldedValue work but the second set
without assignment don't. I asked for help on this at #python (I love
those people in there!) and was told the following...
generatorFuncti on() is a call (obvious) when calling the second set, I
am resetting the iteration and this explains why I only and always get
item1.

ok. *but* why in the world does the first set of calls work?
technically, isn't yieldedValue == generatorFuncti on() on a name basis?
I mean isn't the following technically the same?

generatorFuncti on()
yieldedValue = generatorFuncti on()

aren't they both the same? To me they should be but obviously this
creates the point of this paradox. I don't understand what is happening
here... Can someone care to explain why the assignment works but not
the direct call? In a sense shouldn't the assignment yield the same
results as the direct call and vice versa? I am confused :(

Thank you for any help on this!

[Felipe Almeida Lessa]

Re: My Generator Paradox!

Em Qui, 2006-03-16 às 16:17 -0800, vbgunz escreveu:[color=blue]
> print generatorFuncti on() # <generator object at 0xb723022c>
> print generatorFuncti on().next() # item1
> print generatorFuncti on().next() # item1
> print generatorFuncti on().next() # item1[/color]

Each time you say "generatorFunct ion()", it gives you a new generator,
thus returning the first item again.

[Michal Kwiatkowski]

Re: My Generator Paradox!

vbgunz wrote:[color=blue]
> def generatorFuncti on(sequence=['item1', 'item2', 'item3']):
> for item in sequence:
> yield item
>
> yieldedValue = generatorFuncti on()[/color]

You're creating an iterator here and binding it to name yieldedValue
(which is bogus, it should be named valueGenerator or sth like that).
[color=blue]
> '''this seems to work perfectly.'''
> print '-' * 32
> print yieldedValue # <generator object at 0xb723014c>
> print yieldedValue.ne xt() # item1
> print yieldedValue.ne xt() # item2
> print yieldedValue.ne xt() # item3[/color]

You're calling your iterator's next() method getting all of values, as
expected.
[color=blue]
> '''this is where things don't make any sense!'''
> print '-' * 32
> print generatorFuncti on() # <generator object at 0xb723022c>[/color]
You're creating a new iterator here.
[color=blue]
> print generatorFuncti on().next() # item1[/color]
Another anonymous iterator gets created here. Instantly its next()
method is called, yielding first value.
[color=blue]
> print generatorFuncti on().next() # item1[/color]
And so on...
[color=blue]
> generatorFuncti on() is a call (obvious) when calling the second set, I
> am resetting the iteration and this explains why I only and always get
> item1.
>
> ok. *but* why in the world does the first set of calls work?
> technically, isn't yieldedValue == generatorFuncti on() on a name
> basis? I mean isn't the following technically the same?
>
> generatorFuncti on()
> yieldedValue = generatorFuncti on()[/color]

Well, first statement creates new iterator which is garbage collected
right away (as it has no bindings). Second statement creates an
iterator and binds it to name yieldedValue. Then it can be used as
typical iterator. Calling yieldedValue.ne xt() just calls method of the
same iterator you've created a moment ago. It's still the same object.
The difference is like the difference between following two lines:

list() # creating a new list
new_list = list() # creating a new list and binding its name

So, rewriting your example from generator to dictionary objects:

alist = [1, 2, 3, 4, 5]
print alist # [1, 2, 3, 4, 5]
print alist.pop() # 5
print alist.pop() # 4
print alist.pop() # 3

print [1, 2, 3, 4, 5] # [1, 2, 3, 4, 5]
print [1, 2, 3, 4, 5].pop() # 5
print [1, 2, 3, 4, 5].pop() # 5
print [1, 2, 3, 4, 5].pop() # 5

Remember that generator is an object and you'll be fine.

mk
--
. o . >> http://joker.linuxstuff.pl <<
. . o It's easier to get forgiveness for being wrong
o o o than forgiveness for being right.

[David Wahler]

Re: My Generator Paradox!

vbgunz wrote:[color=blue]
> I am afraid that this is the first time in which I would probably need
> something explained to me as if I were a little child. I am having a
> hard time getting this through my thick skull. What in the world is
> wrong with this!?
>
> ''' ############### ############### ############### ############## '''
>
> def generatorFuncti on(sequence=['item1', 'item2', 'item3']):
> for item in sequence:
> yield item
>
> yieldedValue = generatorFuncti on()
>
> '''this seems to work perfectly.'''
> print '-' * 32
> print yieldedValue # <generator object at 0xb723014c>
> print yieldedValue.ne xt() # item1
> print yieldedValue.ne xt() # item2
> print yieldedValue.ne xt() # item3
>
> '''this is where things don't make any sense!'''
> print '-' * 32
> print generatorFuncti on() # <generator object at 0xb723022c>
> print generatorFuncti on().next() # item1
> print generatorFuncti on().next() # item1
> print generatorFuncti on().next() # item1
>
> ''' ############### ############### ############### ############## '''
>
> the first set of calls assigned to yieldedValue work but the second set
> without assignment don't. I asked for help on this at #python (I love
> those people in there!) and was told the following...
> generatorFuncti on() is a call (obvious) when calling the second set, I
> am resetting the iteration and this explains why I only and always get
> item1.
>
> ok. *but* why in the world does the first set of calls work?
> technically, isn't yieldedValue == generatorFuncti on() on a name basis?
> I mean isn't the following technically the same?
>
> generatorFuncti on()
> yieldedValue = generatorFuncti on()
>
> aren't they both the same?[/color]
[snip]

In that short example, they happen to be different, but equivalent,
objects, but that will not always be the case. Consider this:
[color=blue][color=green][color=darkred]
>>> a = generatorFuncti on()
>>> b = generatorFuncti on()
>>> print a.next()[/color][/color][/color]
item1

Now a and b are both generators for generatorFuncti on, but calling
a.next() again will return 'item2', while calling b.next() will return
'item1'. The value returned by generatorFuncti on is an object, which
has an internal state that makes it distinct from other objects of the
same type. So once your yieldedValue has been altered by calling the
next() method, it is no longer equivalent to a fresh instance of the
generator.

-- David

[Schüle Daniel]

Re: My Generator Paradox!

it's easy to explain

class X:
pass

x=X()
y=X()

x and y are different instances
one can put in x
x.item = 1
y doesn't even have an attribute item for example

similar with generators
they are *different* objects of same kind generator
[color=blue][color=green][color=darkred]
>>> def fib():[/color][/color][/color]
.... a,b = 1,1
.... while True:
.... a,b = b,a+b
.... yield a,b
....[color=blue][color=green][color=darkred]
>>> f1 = fib()
>>> f2 = fib()
>>> f1[/color][/color][/color]
<generator object at 0x4042866c>[color=blue][color=green][color=darkred]
>>> f2[/color][/color][/color]
<generator object at 0x404db42c> # different addresses[color=blue][color=green][color=darkred]
>>> f1 is f2[/color][/color][/color]
False[color=blue][color=green][color=darkred]
>>> f1.next()[/color][/color][/color]
(1, 2)[color=blue][color=green][color=darkred]
>>> f1.next()[/color][/color][/color]
(2, 3)[color=blue][color=green][color=darkred]
>>> f1.next()[/color][/color][/color]
(3, 5)[color=blue][color=green][color=darkred]
>>>
>>>
>>> f2.next()[/color][/color][/color]
(1, 2)[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

it's only natural that each objects starts it's own fibonaci serie

hth, Daniel

[mensanator@aol.com]

Re: My Generator Paradox!


vbgunz wrote:[color=blue]
> I am afraid that this is the first time in which I would probably need
> something explained to me as if I were a little child. I am having a
> hard time getting this through my thick skull. What in the world is
> wrong with this!?
>
> ''' ############### ############### ############### ############## '''
>
> def generatorFuncti on(sequence=['item1', 'item2', 'item3']):
> for item in sequence:
> yield item
>
> yieldedValue = generatorFuncti on()
>
> '''this seems to work perfectly.'''
> print '-' * 32
> print yieldedValue # <generator object at 0xb723014c>
> print yieldedValue.ne xt() # item1
> print yieldedValue.ne xt() # item2
> print yieldedValue.ne xt() # item3
>
> '''this is where things don't make any sense!'''
> print '-' * 32
> print generatorFuncti on() # <generator object at 0xb723022c>
> print generatorFuncti on().next() # item1
> print generatorFuncti on().next() # item1
> print generatorFuncti on().next() # item1
>
> ''' ############### ############### ############### ############## '''
>
> the first set of calls assigned to yieldedValue work but the second set
> without assignment don't. I asked for help on this at #python (I love
> those people in there!) and was told the following...
> generatorFuncti on() is a call (obvious) when calling the second set, I
> am resetting the iteration and this explains why I only and always get
> item1.
>
> ok. *but* why in the world does the first set of calls work?
> technically, isn't yieldedValue == generatorFuncti on() on a name basis?
> I mean isn't the following technically the same?
>
> generatorFuncti on()
> yieldedValue = generatorFuncti on()[/color]

No. Look at this
[color=blue][color=green][color=darkred]
>>> a = generatorFuncti on()
>>> b = generatorFuncti on()
>>> a==b[/color][/color][/color]
False

Why aren'y they the same? Here's a clue:
[color=blue][color=green][color=darkred]
>>> generatorFuncti on()[/color][/color][/color]
<generator object at 0x00B28418>[color=blue][color=green][color=darkred]
>>> generatorFuncti on()[/color][/color][/color]
<generator object at 0x00AD24E0>

Note the addresses are different.

Try this
[color=blue][color=green][color=darkred]
>>> b.next()[/color][/color][/color]
'item1'[color=blue][color=green][color=darkred]
>>> b.next()[/color][/color][/color]
'item2'[color=blue][color=green][color=darkred]
>>> generatorFuncti on().next()[/color][/color][/color]
'item1'

Just like your example, the generator re-initailized and
printed item1. But that's not the same generator as b
[color=blue][color=green][color=darkred]
>>> b.next()[/color][/color][/color]
'item3'

[color=blue]
>
> aren't they both the same? To me they should be but obviously this
> creates the point of this paradox. I don't understand what is happening
> here... Can someone care to explain why the assignment works but not
> the direct call? In a sense shouldn't the assignment yield the same
> results as the direct call and vice versa? I am confused :(
>
> Thank you for any help on this![/color]

[Fredrik Lundh]

Re: My Generator Paradox!

"vbgunz" wrote:
[color=blue]
> I am afraid that this is the first time in which I would probably need
> something explained to me as if I were a little child. I am having a
> hard time getting this through my thick skull. What in the world is
> wrong with this!?
>
> ''' ############### ############### ############### ############## '''
>
> def generatorFuncti on(sequence=['item1', 'item2', 'item3']):
> for item in sequence:
> yield item
>
> yieldedValue = generatorFuncti on()
>
> '''this seems to work perfectly.'''
> print '-' * 32
> print yieldedValue # <generator object at 0xb723014c>
> print yieldedValue.ne xt() # item1
> print yieldedValue.ne xt() # item2
> print yieldedValue.ne xt() # item3
>
> '''this is where things don't make any sense!'''
> print '-' * 32
> print generatorFuncti on() # <generator object at 0xb723022c>
> print generatorFuncti on().next() # item1
> print generatorFuncti on().next() # item1
> print generatorFuncti on().next() # item1
>
> ''' ############### ############### ############### ############## '''[/color]

does the following surprise you too ?

f = open("filename" )
print f.readline() # prints first line
print f.readline() # prints second line
print f.readline() # prints third line

print open("filename" ).readline() # prints first line
print open("filename" ).readline() # prints first line
print open("filename" ).readline() # prints first line

</F>

[vbgunz]

Re: My Generator Paradox!

I believe I understand now. the yield keyword is sort of like a cousin
to return. return will bring back an object I can work with and so does
yield *but* yield's object will most likely support the .next() method.

So, if I worked with a function that ends with the return keyword and
it returns a list, I can run list operations and list methods on it. if
a function ends with the yield keyword a generator should return.

So, calling the function by it's name will always reset and initialize
the generator. Whereas assigning to the functions yielded return grants
access to the real generator in which I can use the next() method.

Maybe I've explained it wrong *but* it does make sense to me now. I
just couldn't grasp it because I am still new to the keyword yield and
didn't know it sort of works like return.

I really wish to thank you fellas so much for your examples and
explanations! I think I got it! I thank you all again!

[Robert Kern]

Re: My Generator Paradox!

vbgunz wrote:[color=blue]
> I believe I understand now. the yield keyword is sort of like a cousin
> to return. return will bring back an object I can work with and so does
> yield *but* yield's object will most likely support the .next() method.[/color]

No, that's not really how it works. When a generator function is called, it
returns the generator object immediately. None of the code inside is executed.
Every time you call that generator function, you get a new generator object with
the initial state. The objects that are yielded inside the code don't show up yet.

The code inside the generator gets executed only when the generator object is
iterated over (or its .next() method is called). The objects that are yielded
are the results of calling the .next() method.

--
Robert Kern
robert.kern@gma il.com

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Ron Adam]

Re: My Generator Paradox!

Robert Kern wrote:[color=blue]
> vbgunz wrote:[color=green]
>> I believe I understand now. the yield keyword is sort of like a cousin
>> to return. return will bring back an object I can work with and so does
>> yield *but* yield's object will most likely support the .next() method.[/color]
>
> No, that's not really how it works. When a generator function is called, it
> returns the generator object immediately. None of the code inside is executed.
> Every time you call that generator function, you get a new generator object with
> the initial state. The objects that are yielded inside the code don't show up yet.
>
> The code inside the generator gets executed only when the generator object is
> iterated over (or its .next() method is called). The objects that are yielded
> are the results of calling the .next() method.
>[/color]

Maybe this will clarify it further.

[color=blue][color=green][color=darkred]
>>> def gen(n):[/color][/color][/color]
.... while 1:
.... print 'before yield'
.... yield n
.... print 'after yield'
....[color=blue][color=green][color=darkred]
>>> g = gen('hello')
>>> g.next()[/color][/color][/color]
before yield
'hello'[color=blue][color=green][color=darkred]
>>> g.next()[/color][/color][/color]
after yield
before yield
'hello'[color=blue][color=green][color=darkred]
>>> g.next()[/color][/color][/color]
after yield
before yield
'hello'

When the next() method is called the generator runs until it reaches a
yield. At which point it's rests until the next() method is called again.

Although there are times when I wish it could run (as a thread) until it
reaches a yield and then continue after the next() method is called
until it reaches the next yield.

Cheers,
Ron

[vbgunz]

Re: My Generator Paradox!

OK. I hope my understanding of the yield keyword and generators in a
general sense are now better understood. When a generator function is
assigned to an identifier, no code is executed and a generator is
immediately returned. When the next() method is called on the new
generator, code from top to bottom executes within the generator until
it reaches it's first yield. Many yields can appear within one
generator. When this is the case a next method call will execute code
from yield to yield. Code that appears in a loop after a yield keyword
is executed on the next() method call.

I hope I got it right. I love you guys for your patience and examples.
It is greatly appreciated and means very much to me! Thank you fellas!

[Ron Adam]

Re: My Generator Paradox!

vbgunz wrote:[color=blue]
> OK. I hope my understanding of the yield keyword and generators in a
> general sense are now better understood. When a generator function is
> assigned to an identifier, no code is executed and a generator is
> immediately returned. When the next() method is called on the new
> generator, code from top to bottom executes within the generator until
> it reaches it's first yield. Many yields can appear within one
> generator. When this is the case a next method call will execute code
> from yield to yield. Code that appears in a loop after a yield keyword
> is executed on the next() method call.
>
> I hope I got it right. I love you guys for your patience and examples.
> It is greatly appreciated and means very much to me! Thank you fellas![/color]

Yep, looks like you have it. ;-)

Only need to add what happens if a generator exits the end after the
yield(s).

If it were a function it would return a None object even if it didn't
have a return at the end. But a generator raises a StopIteration
Exception.
[color=blue][color=green][color=darkred]
>>> def gen():[/color][/color][/color]
.... yield 'hello'
.... print 'all done'
....[color=blue][color=green][color=darkred]
>>> g = gen()
>>> g.next()[/color][/color][/color]
'hello'[color=blue][color=green][color=darkred]
>>> g.next()[/color][/color][/color]
all done
Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration

This is the signal to indicate iteration is finished. You don't see it
when you are using generators as iterators because it's usually caught
by the object or statement using the generator.

Cheers,
Ron