Question about exausted iterators

[Christophe]

Is there a good reason why when you try to take an element from an
already exausted iterator, it throws StopIteration instead of some other
exception ? I've lost quite some times already because I was using a lot
of iterators and I forgot that that specific function parameter was one.

Exemple :
[color=blue][color=green][color=darkred]
>>> def f(i):[/color][/color][/color]
.... print list(i)
.... print list(i)
....[color=blue][color=green][color=darkred]
>>> f(iter(range(2) ))[/color][/color][/color]
[0, 1]
[][color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

This is using Python 2.4.2

[Terry Reedy]

Re: Question about exausted iterators


"Christophe " <chris.cavalari a@free.fr> wrote in message
news:446b48e2$0 $5293$626a54ce@ news.free.fr...[color=blue]
> Is there a good reason why when you try to take an element from an
> already exausted iterator, it throws StopIteration instead of some other
> exception ?[/color]

Yes.
..
..
To distinguish the control message "I am done yielding values, as per the
code specification (so don't bother calling me again)." from error messages
that say "Something is wrong, I cannot yield values and give up." In other
words, to distinguish expected correct behavior from unexpected incorrect
behavior. This is essential for the normal and correct use of iterators.
[color=blue]
> I've lost quite some times already because I was using a lot
> of iterators and I forgot that that specific function parameter was one.[/color]

I think you mean 'specific function argument' for a parameter which could
be any iterable.
[color=blue]
> Exemple :[/color]
Example
[color=blue][color=green][color=darkred]
> >>> def f(i):[/color][/color]
> ... print list(i)
> ... print list(i)
> ...[color=green][color=darkred]
> >>> f(iter(range(2) ))[/color][/color]
> [0, 1]
> [][/color]

As per specification.
I am guessing that you want the first list() call to terminate normally and
return a list, which requires exhausted i to raise StopIteration, while you
want the second list() to not terminate but raise an exception, which
requires exhausted i to raise something other than StopIteration. Tough.

One solution is call list(i) exactly once:

def f(i):
li = list(i)
print li
print li

Another is to document f as requiring that i be a non-iterator reiterable
iterable and only pass correct arguments.

A third is to add a line like
if iter(i) is i: raise TypeError("inpu t appears to be iterator")

This is not quite exact since it will improperly exclude self-iterator
reiterables (which, I believe, no builtin is) and improperly pass
non-reiterable non-iterator iterables (at least some file objects). But it
might work for all your cases.

Terry Jan Reedy

[George Sakkis]

Re: Question about exausted iterators

Christophe wrote:
[color=blue]
> Is there a good reason why when you try to take an element from an
> already exausted iterator, it throws StopIteration instead of some other
> exception ? I've lost quite some times already because I was using a lot
> of iterators and I forgot that that specific function parameter was one.
>
> Exemple :
>[color=green][color=darkred]
> >>> def f(i):[/color][/color]
> ... print list(i)
> ... print list(i)
> ...[color=green][color=darkred]
> >>> f(iter(range(2) ))[/color][/color]
> [0, 1]
> [][/color]

Whether trying to iterate over an exhausted iterator should be treated
differently is appication dependent. In most cases, you don't really
care to distinguish between an iterator that yields no elements and an
iterator that did yield some elements before but it has been exhausted.
If you do care, you can roll your own iterator wrapper:


class ExhaustibleIter ator(object):
def __init__(self, iterable):
self._next = getattr(iterabl e, 'next', iter(iterable). next)
self._exhausted = False

def next(self):
if self._exhausted :
raise ExhaustedIterat orException()
try: return self._next()
except StopIteration:
self._exhausted = True
raise

def __iter__(self):
return self

class ExhaustedIterat orException(Exc eption):
pass


And then in your function:
def f(i):
i = ExhaustibleIter ator(i)
print list(i)
print list(i)


HTH,
George

[Christophe]

Re: Question about exausted iterators

Terry Reedy a écrit :[color=blue]
> "Christophe " <chris.cavalari a@free.fr> wrote in message
> news:446b48e2$0 $5293$626a54ce@ news.free.fr...
>[color=green]
>>Is there a good reason why when you try to take an element from an
>>already exausted iterator, it throws StopIteration instead of some other
>>exception ?[/color]
>
>
> Yes.
> .
> .
> To distinguish the control message "I am done yielding values, as per the
> code specification (so don't bother calling me again)." from error messages
> that say "Something is wrong, I cannot yield values and give up." In other
> words, to distinguish expected correct behavior from unexpected incorrect
> behavior. This is essential for the normal and correct use of iterators.[/color]

You talk about expected behaviour and my expected behaviour is that an
iterator should not be usable once it has raised StopIteration once.
[color=blue][color=green][color=darkred]
>>>>>def f(i):[/color]
>>
>>... print list(i)
>>... print list(i)
>>...
>>[color=darkred]
>>>>>f(iter(ran ge(2)))[/color]
>>
>>[0, 1]
>>[][/color]
>
>
> As per specification.[/color]

Specifications sometimes have "bugs" too.
[color=blue]
> I am guessing that you want the first list() call to terminate normally and
> return a list, which requires exhausted i to raise StopIteration, while you
> want the second list() to not terminate but raise an exception, which
> requires exhausted i to raise something other than StopIteration. Tough.[/color]

Exactly. This would be a sane way to handle it.
[color=blue]
> One solution is call list(i) exactly once:
>
> def f(i):
> li = list(i)
> print li
> print li[/color]

Ok, call me stupid if you want but I know perfectly well the "solution"
to that problem ! Come on, I was showing example code of an horrible
gotcha on using iterators.





Instead of saying that all works as intended could you be a little
helpful and tell me why it was intended in such an obviously broken way
instead ?

[looping]

Re: Question about exausted iterators


Christophe wrote:[color=blue]
> Ok, call me stupid if you want but I know perfectly well the "solution"
> to that problem ! Come on, I was showing example code of an horrible
> gotcha on using iterators.
>[/color]

OK, your are stupid ;-)
Why asking questions when you don't want to listen answers ?

[color=blue]
>
>
> Instead of saying that all works as intended could you be a little
> helpful and tell me why it was intended in such an obviously broken way
> instead ?[/color]

Why an exausted iterator must return an Exception (other than
StopIteration of course) ?
Well an exausted iterator could be seen like an empty string or an
empty list (or tons of others things), so you expect the code
for car in "":
print car
to return an Exception because it's empty ???
It's your job to check the iterator when it need to be.

Regards.
Dom

[Christophe]

Re: Question about exausted iterators

looping a écrit :[color=blue]
> Christophe wrote:
>[color=green]
>>Ok, call me stupid if you want but I know perfectly well the "solution"
>>to that problem ! Come on, I was showing example code of an horrible
>>gotcha on using iterators.
>>[/color]
>
>
> OK, your are stupid ;-)
> Why asking questions when you don't want to listen answers ?[/color]

Because I'm still waiting for a valid answer to my question. The answer
"Because it has been coded like that" or is not a valid one.
[color=blue][color=green]
>>Instead of saying that all works as intended could you be a little
>>helpful and tell me why it was intended in such an obviously broken way
>>instead ?[/color]
>
> Why an exausted iterator must return an Exception (other than
> StopIteration of course) ?[/color]

Because it's exausted. Because it has been for me a frequent cause of
bugs and because I have yet to see a valid use case for such behaviour.
[color=blue]
> Well an exausted iterator could be seen like an empty string or an
> empty list (or tons of others things), so you expect the code
> for car in "":
> print car
> to return an Exception because it's empty ???[/color]

Of course not.
[color=blue]
> It's your job to check the iterator when it need to be.[/color]

It's my job to avoid coding bugs, it's the language job to avoid placing
pitfalls everywhere I go.



I must confess I have a strong opinion on that point. Not long ago I
started working on some fresh code where I decided to use a lot of
iterators and set instead of list if possible. That behaviour has caused
me to lose quite some time tracking bugs.

[Fredrik Lundh]

Re: Question about exausted iterators

Christophe wrote:
[color=blue]
> Because I'm still waiting for a valid answer to my question. The answer
> "Because it has been coded like that" or is not a valid one.[/color]

it's been coded like that because that's what the specification says:

http://www.python.org/dev/peps/pep-0234/

</F>

[Christophe]

Re: Question about exausted iterators

Fredrik Lundh a écrit :[color=blue]
> Christophe wrote:
>[color=green]
>> Because I'm still waiting for a valid answer to my question. The
>> answer "Because it has been coded like that" or is not a valid one.[/color]
>
>
> it's been coded like that because that's what the specification says:
>
> http://www.python.org/dev/peps/pep-0234/[/color]

I didn't though I had to mention that "Because the spec has been writen
like that" wasn't a valid answer either.

[Fredrik Lundh]

Re: Question about exausted iterators

Christophe wrote:
[color=blue][color=green][color=darkred]
>>> Because I'm still waiting for a valid answer to my question. The
>>> answer "Because it has been coded like that" or is not a valid one.[/color]
>>
>> it's been coded like that because that's what the specification says:
>>
>> http://www.python.org/dev/peps/pep-0234/[/color]
>
> I didn't though I had to mention that "Because the spec has been writen
> like that" wasn't a valid answer either.[/color]

so what is a valid answer?

</F>

[Diez B. Roggisch]

Re: Question about exausted iterators

Christophe wrote:
[color=blue]
> Fredrik Lundh a écrit :[color=green]
>> Christophe wrote:
>>[color=darkred]
>>> Because I'm still waiting for a valid answer to my question. The
>>> answer "Because it has been coded like that" or is not a valid one.[/color]
>>
>>
>> it's been coded like that because that's what the specification says:
>>
>> http://www.python.org/dev/peps/pep-0234/[/color]
>
> I didn't though I had to mention that "Because the spec has been writen
> like that" wasn't a valid answer either.[/color]

The important thing is: it _is_ specified. And what about code like this:


iterable = produce_some_it erable()

for item in iterable:
if some_condition( item)
break
do_something()

for item in iterable:
do_something_wi th_the_rest()


If it weren't for StopIteration raised if the iterable was exhausted, you'd
have to clutter that code with something like

try:
for item in iterable:
do_something_wi th_the_rest()
except IteratorExhaust ed:
pass

What makes you say that this is better than the above? Just because _you_
had some cornercases that others seems not to have (at least that
frequently, I personally can't remember I've ever bitten by it) isn't a
valid reason to _not_ do it as python does.

Besides that: it would be a major change of semantics of iterators that I
seriously doubt it would make it into anything before P3K. So - somewhat a
moot point to discuss here I'd say.

Diez

[Christophe]

Re: Question about exausted iterators

Fredrik Lundh a écrit :[color=blue]
> Christophe wrote:
>[color=green][color=darkred]
>>>> Because I'm still waiting for a valid answer to my question. The
>>>> answer "Because it has been coded like that" or is not a valid one.
>>>
>>>
>>> it's been coded like that because that's what the specification says:
>>>
>>> http://www.python.org/dev/peps/pep-0234/[/color]
>>
>>
>> I didn't though I had to mention that "Because the spec has been
>> writen like that" wasn't a valid answer either.[/color]
>
>
> so what is a valid answer?[/color]

Some valid use case for that behaviour, some example of why what I ask
could cause problems, some implementation difficulties etc ...

Saying it's like that because someone said so isn't exactly what I was
expecting as an answer :) People sometimes can be wrong you know.

[Roel Schroeven]

Re: Question about exausted iterators

Fredrik Lundh schreef:[color=blue]
> Christophe wrote:
>[color=green][color=darkred]
>>>> Because I'm still waiting for a valid answer to my question.
>>>> The answer "Because it has been coded like that" or is not a
>>>> valid one.
>>> it's been coded like that because that's what the specification
>>> says:
>>>
>>> http://www.python.org/dev/peps/pep-0234/[/color]
>> I didn't though I had to mention that "Because the spec has been
>> writen like that" wasn't a valid answer either.[/color]
>
> so what is a valid answer?[/color]

I think he wants to know why the spec has been written that way.

The rationale mentions exhausted iterators:

"Once a particular iterator object has raised StopIteration, will
it also raise StopIteration on all subsequent next() calls?
Some say that it would be useful to require this, others say
that it is useful to leave this open to individual iterators.
Note that this may require an additional state bit for some
iterator implementations (e.g. function-wrapping iterators).

Resolution: once StopIteration is raised, calling it.next()
continues to raise StopIteration."

This doesn't, however, completey answer the OP's question, I think. It
is about raising or not raising StopIteration on subsequent next() calls
but doesn't say anything on possible alternatives, such as raising
another exception (I believe that's what the OP would like).

Not that I know of use cases for other exceptions after StopIteration;
just clarifying what I think the OP means.

--
If I have been able to see further, it was only because I stood
on the shoulders of giants. -- Isaac Newton

Roel Schroeven

[Christophe]

Re: Question about exausted iterators

Diez B. Roggisch a écrit :[color=blue]
> Christophe wrote:
>
>[color=green]
>>Fredrik Lundh a écrit :
>>[color=darkred]
>>>Christophe wrote:
>>>
>>>
>>>>Because I'm still waiting for a valid answer to my question. The
>>>>answer "Because it has been coded like that" or is not a valid one.
>>>
>>>
>>>it's been coded like that because that's what the specification says:
>>>
>>> http://www.python.org/dev/peps/pep-0234/[/color]
>>
>>I didn't though I had to mention that "Because the spec has been writen
>>like that" wasn't a valid answer either.[/color]
>
>
> The important thing is: it _is_ specified. And what about code like this:
>
>
> iterable = produce_some_it erable()
>
> for item in iterable:
> if some_condition( item)
> break
> do_something()
>
> for item in iterable:
> do_something_wi th_the_rest()
>
>
> If it weren't for StopIteration raised if the iterable was exhausted, you'd
> have to clutter that code with something like
>
> try:
> for item in iterable:
> do_something_wi th_the_rest()
> except IteratorExhaust ed:
> pass[/color]

It would be ugly but you could do that instead :

iterable = produce_some_it erable()

for item in iterable:
if some_condition( item)
break
do_something()
else:
iterable = []

for item in iterable:
do_something_wi th_the_rest()

I'll admit that the else clause in for/while loops isn't the most common
and so some people might be a little troubled by that.

There's also that :

iterable = produce_some_it erable()

for item in iterable:
if some_condition( item)
for item in iterable:
do_something_wi th_the_rest()
break
do_something()
[color=blue]
> What makes you say that this is better than the above? Just because _you_
> had some cornercases that others seems not to have (at least that
> frequently, I personally can't remember I've ever bitten by it) isn't a
> valid reason to _not_ do it as python does.[/color]

Maybe I've used more iterables than most of you. Maybe I've been doing
that wrong. But I'd like to think that if I've made those mistakes,
others will make it too and would benefit for some help in debugging
that from the interpreter :)
[color=blue]
> Besides that: it would be a major change of semantics of iterators that I
> seriously doubt it would make it into anything before P3K. So - somewhat a
> moot point to discuss here I'd say.[/color]

It wouldn't be such a big semantic change I think. You could add that
easily[1] as deprecation warning at first and later on switch to a full
blown error.

[1] "Easily" provided you can easily code what I ask itself ;)

[Christophe]

Re: Question about exausted iterators

Roel Schroeven a écrit :[color=blue]
> Fredrik Lundh schreef:[color=green]
>> so what is a valid answer?[/color]
>
>
> I think he wants to know why the spec has been written that way.
>
> The rationale mentions exhausted iterators:
>
> "Once a particular iterator object has raised StopIteration, will
> it also raise StopIteration on all subsequent next() calls?
> Some say that it would be useful to require this, others say
> that it is useful to leave this open to individual iterators.
> Note that this may require an additional state bit for some
> iterator implementations (e.g. function-wrapping iterators).
>
> Resolution: once StopIteration is raised, calling it.next()
> continues to raise StopIteration."
>
> This doesn't, however, completey answer the OP's question, I think. It
> is about raising or not raising StopIteration on subsequent next() calls
> but doesn't say anything on possible alternatives, such as raising
> another exception (I believe that's what the OP would like).[/color]

Exactly !
[color=blue]
> Not that I know of use cases for other exceptions after StopIteration;
> just clarifying what I think the OP means.[/color]

There are no use cases yet for me. I want those exceptions as an hard
error for debuging purposes.

[Fredrik Lundh]

Re: Question about exausted iterators

Christophe wrote:
[color=blue]
> Maybe I've used more iterables than most of you. Maybe I've been doing
> that wrong.[/color]

your problem is that you're confusing iterables with sequences. they're
two different things.

</F>