Why are "broken iterators" broken?

[Steven D'Aprano]

According to the Python docs, once an iterator raises StopIteration, it
should continue to raise StopIteration forever. Iterators that fail to
behave in this fashion are deemed to be "broken":

http://docs.python.org/lib/typeiter.html

I don't understand the reasoning behind this. As I understand it, an
iterator is something like a stream. There's no constraint that once a
stream is empty it must remain empty forever.

Can somebody explain why "broken iterators" are broken?


--
Steven

[Fredrik Lundh]

Re: Why are "broken iterators" broken?

Steven D'Aprano wrote:

According to the Python docs, once an iterator raises StopIteration, it
should continue to raise StopIteration forever. Iterators that fail to
behave in this fashion are deemed to be "broken":
>

http://docs.python.org/lib/typeiter.html

>
I don't understand the reasoning behind this. As I understand it, an
iterator is something like a stream. There's no constraint that once a
stream is empty it must remain empty forever.

it's a design guideline, not an absolute rule.

but I disagree that an iterator is "something like a stream". it's
rather "something like a pointer or an index", that is, an object that
helps you iterate over all members in a collection.

</F>

[Roy Smith]

Re: Why are &quot;broken iterators&quot; broken?

In article <mailman.1327.1 222010852.3487. python-list@python.org >,
Fredrik Lundh <fredrik@python ware.comwrote:

Steven D'Aprano wrote:
>

According to the Python docs, once an iterator raises StopIteration, it
should continue to raise StopIteration forever. Iterators that fail to
behave in this fashion are deemed to be "broken":

http://docs.python.org/lib/typeiter.html

I don't understand the reasoning behind this. As I understand it, an
iterator is something like a stream. There's no constraint that once a
stream is empty it must remain empty forever.

>
it's a design guideline, not an absolute rule.
>
but I disagree that an iterator is "something like a stream". it's
rather "something like a pointer or an index", that is, an object that
helps you iterate over all members in a collection.
>
</F>

There are plausible examples of collections which grow while you're
iterating over them. I'm thinking specifically of a queue in a
multi-threaded application. One thread pushes work onto the back of the
queue while another pops from the front. The queue could certainly go
empty at times. But, maybe a Python iterator is just the wrong way to
model such behavior.

[Fredrik Lundh]

Re: Why are &quot;broken iterators&quot; broken?

Roy Smith wrote:

There are plausible examples of collections which grow while you're
iterating over them. I'm thinking specifically of a queue in a
multi-threaded application. One thread pushes work onto the back of the
queue while another pops from the front. The queue could certainly go
empty at times. But, maybe a Python iterator is just the wrong way to
model such behavior.

you probably want the consumer thread to block when it catches up with
the producer, rather than exit.

(that's the default behaviour of Python's Queue object, btw)

</F>

[Terry Reedy]

Re: Why are &quot;broken iterators&quot; broken?

Steven D'Aprano wrote:

According to the Python docs, once an iterator raises StopIteration, it
should continue to raise StopIteration forever. Iterators that fail to
behave in this fashion are deemed to be "broken":
>

http://docs.python.org/lib/typeiter.html

>
I don't understand the reasoning behind this. As I understand it, an
iterator is something like a stream. There's no constraint that once a
stream is empty it must remain empty forever.

It is quite possible that a stream reader will return '' on one call and
then something non-empty the next. An iterator that reads a stream
and yields chunks of whatever size should either block until it gets
sufficient data or yield nulls as long as the stream is open and not
raise StopIteration until the steam is closed and it has yielded the
last chunk of data.

Can somebody explain why "broken iterators" are broken?

There is an important different between a store that is closed until the
next day and one that closed - out of business. Similarly, there is a
difference between an item being out-of-stock until the next delivery
and out-of-stock and discontinued permanently, or between a road closed
for repairs versus removal for something else. Using the same sign or
signal for temporary and permanent conditions is confusing and therefore
'broken'.

Terry Jan Reedy

[Miles]

Re: Why are &quot;broken iterators&quot; broken?

On Sun, Sep 21, 2008 at 11:13 AM, Steven D'Aprano
<steve@remove-this-cybersource.com .auwrote:

According to the Python docs, once an iterator raises StopIteration, it
should continue to raise StopIteration forever. Iterators that fail to
behave in this fashion are deemed to be "broken":
>

http://docs.python.org/lib/typeiter.html

>
I don't understand the reasoning behind this. As I understand it, an
iterator is something like a stream. There's no constraint that once a
stream is empty it must remain empty forever.
>
Can somebody explain why "broken iterators" are broken?

It's not a terribly onerous restriction. If you're iterating with a
for-loop, you can make the iterable return a new iterator object when
the old one is exhausted, and if the intent is for the next()-method
to be called directly, you don't have to conform to the iterator
protocol.

Strictly speaking, file objects are broken iterators:

Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:16)

>>f = open('foo')
>>it = iter(f)
>>it.next()

'hi\n'

>>it.next()

'bye\n'

>>it.next()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

>>f.seek(0)
>>it.next()

'hi\n'

-Miles

[Craig Allen]

Re: Why are &quot;broken iterators&quot; broken?

I'm interested what others think of this because at first I couldn't
get it... I have an object which can iterate over its parts... and at
first I thought, what? I'm supposed to create a new object every time
the user needs to iterate the contents?

In the end I interpreted that statement as if "unless __iter__()" is
called again, in which case it makes sense that an iterator should
stay "finished" until it's told to start to iterate again.

Then even the stream analogy holds, you don't expect a stream to say
EOF then start giving you bytes (from beyond the end, I guess)...
instead, such a stream would more likely block if it got to the end of
available data.

I'm probably being stupid in this... perhaps I'm the only one that at
first interpreted the phrase as being regardless of a fresh call to
__iter__()... and that the OP was worried about some other
implication. But if I was wrong to think it's ok for one and the same
iterator to reset when __iter__() is called again, then I seriously
don't understand.

what does "forever" mean in that dictum?

[greg]

Re: Why are &quot;broken iterators&quot; broken?

Craig Allen wrote:

In the end I interpreted that statement as if "unless __iter__()" is
called again

It seems you were confusing iterators and iterables. The
iterator is the object that is returned by calling
__iter__() on an iterable, and yes, you are expected to
get a new one each time you want to start iterating
again.

--
Greg

[Lie]

Re: Why are &quot;broken iterators&quot; broken?

On Sep 21, 10:13 pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com .auwrote:

According to the Python docs, once an iterator raises StopIteration, it
should continue to raise StopIteration forever. Iterators that fail to
behave in this fashion are deemed to be "broken":
>

http://docs.python.org/lib/typeiter.html

>
I don't understand the reasoning behind this. As I understand it, an
iterator is something like a stream. There's no constraint that once a
stream is empty it must remain empty forever.

I think empty != StopIteration. StopIteration (IMHO) shouldn't be
raised when the stream is empty, instead a sentinel value specifying
that "there is no data yet, but if you wait there might be" should be
returned (possibly None or empty string). When you raise
StopIteration, it is a signal that I don't have any more data and
there is no use in waiting.

Can somebody explain why "broken iterators" are broken?
>
--
Steven