Strange Behavior with Old-Style classes and implicit __contains__

[rconradharris@gmail.com]

A co-worker of mine came across some interesting behavior in the
Python interpreter today and I'm hoping someone more knowledgeable in
Python internals can explain this to me.

First, we create an instance of an Old-Style class without defining a
__contains__ but instead define a __getitem__ method in which we raise
KeyError. Next we repeatedly use the 'in' operator to test to see
whether something, a string, an int, etc is an attribute of this new
instance.

Here's the strange part: The first test will return False as if the
__getitem__ was never called. The next test will raise a KeyError as
we'd expect. The test after that will again return False. This goes on
ad infinitum.

In Code:
Python 2.5 (r25:51908, Jan 21 2007, 03:10:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
Type "help", "copyright" , "credits" or "license" for more
information.

>>class Foo:

... def __getitem__(sel f, key):
... raise KeyError
...

>>foo = Foo()
>>"asdf" in foo

False

>>"asdf" in foo

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getitem__
KeyError

>>"asdf" in foo

False

>>"asdf" in foo

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getitem__
KeyError


According to the language reference, if __contains__ isn't defined for
Old-Style classes and __getitem__ is defined, __getitem__ will be
called. So, how then can False ever be returned?

And to make matters worse, I've set up a situation where Python will
flat-out return incorrect results:

Python 2.5 (r25:51908, Jan 21 2007, 03:10:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
Type "help", "copyright" , "credits" or "license" for more information.

>>class Foo:

.... def __getitem__(sel f, key):
.... raise KeyError
....

>>foo = Foo()
>>"asdf" in foo

False

>>1 in set([1,2,3]) <---- So the prior KeyError from another class is interacting and producing bad output

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getitem__
KeyError

According to our cursory testing, this funny business doesn't happen
with New-Style Classes or when using PyPy.

If anyone can provide some insight into this, it would be greatly
appreciated.

Thanks,
Rick Harris

[Steven D'Aprano]

Re: Strange Behavior with Old-Style classes and implicit __contains__

On Wed, 11 Apr 2007 16:37:35 -0700, rconradharris wrote:

A co-worker of mine came across some interesting behavior in the
Python interpreter today and I'm hoping someone more knowledgeable in
Python internals can explain this to me.
>
First, we create an instance of an Old-Style class without defining a
__contains__ but instead define a __getitem__ method in which we raise
KeyError. Next we repeatedly use the 'in' operator to test to see
whether something, a string, an int, etc is an attribute of this new
instance.
>
Here's the strange part: The first test will return False as if the
__getitem__ was never called. The next test will raise a KeyError as
we'd expect. The test after that will again return False. This goes on
ad infinitum.

I can confirm that. It looks like __getitem__ is only being called every
second time.

class Parrot:
def __getitem__(sel f, n):
print "Checking index %s..." % n
raise KeyError

def tester(n):
parrot = Parrot()
results = []
for i in range(n):
try:
results.append( i in parrot)
except KeyError:
results.append( "KeyError")
return results



Here are the results under Python 2.5:

>>tester(10)

Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
[False, 'KeyError', False, 'KeyError', False,
'KeyError', False, 'KeyError', False, 'KeyError']


And here are the results under Python 2.4.3:

>>tester(10)

Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
['KeyError', 'KeyError', 'KeyError', 'KeyError', 'KeyError',
'KeyError', 'KeyError', 'KeyError', 'KeyError', 'KeyError']


Looks like a bug to me.



--
Steven.

[Gabriel Genellina]

Re: Strange Behavior with Old-Style classes and implicit __contains__

En Wed, 11 Apr 2007 20:37:35 -0300, <rconradharris@ gmail.comescrib ió:

First, we create an instance of an Old-Style class without defining a
__contains__ but instead define a __getitem__ method in which we raise
KeyError. Next we repeatedly use the 'in' operator to test to see
whether something, a string, an int, etc is an attribute of this new
instance.
>
Here's the strange part: The first test will return False as if the
__getitem__ was never called. The next test will raise a KeyError as
we'd expect. The test after that will again return False. This goes on
ad infinitum.
>
In Code:
Python 2.5 (r25:51908, Jan 21 2007, 03:10:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
Type "help", "copyright" , "credits" or "license" for more
information.

>>class Foo:

... def __getitem__(sel f, key):
... raise KeyError
...

>>foo = Foo()
>>"asdf" in foo

False

>>"asdf" in foo

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getitem__
KeyError

First I want to say that __getitem__ should raise IndexError, not
KeyError, to indicate "not found" - just to make clear the observed
behavior. (Using IndexError, you always get False, as expected).
Python 2.4 and 2.3 never return False, always showing the KeyError
exception, also as expected.

>>>foo = Foo()
>>>"asdf" in foo

False

>>>1 in set([1,2,3]) <---- So the prior KeyError from another class
>>>is interacting and producing bad output

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getitem__
KeyError

I have a displayhook installed, and it also were interfering with the
results; it appears that the next __getitem__ call after the KeyError was
raised (wherever it is called) "sees" the previous exception.

You should file a bug at http://sourceforge.net/bugs/?group_id=5470

--
Gabriel Genellina

[Steven D'Aprano]

Re: Strange Behavior with Old-Style classes and implicit __contains__

On Thu, 12 Apr 2007 00:32:32 -0300, Gabriel Genellina wrote:

First I want to say that __getitem__ should raise IndexError, not
KeyError, to indicate "not found"

How do you know the Original Poster's class was meant to be a sequence
rather than a mapping?

--
Steven.

[Scott David Daniels]

Re: Strange Behavior with Old-Style classes and implicit __contains__

Steven D'Aprano wrote:

On Wed, 11 Apr 2007 16:37:35 -0700, rconradharris wrote:
...
Here are the results under Python 2.5:
>

>>>tester(10)

Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
Checking index 0...
[False, 'KeyError', False, 'KeyError', False,
'KeyError', False, 'KeyError', False, 'KeyError']
>
>
And here are the results under Python 2.4.3:
>

>>>tester(10)

[works]

>
Looks like a bug to me.

No problem with 2.5.1c1 here.

--Scott David Daniels
scott.daniels@a cm.org

[Gabriel Genellina]

Re: Strange Behavior with Old-Style classes and implicit __contains__

En Thu, 12 Apr 2007 01:23:08 -0300, Steven D'Aprano
<steve@REMOVE.T HIS.cybersource .com.auescribió :

On Thu, 12 Apr 2007 00:32:32 -0300, Gabriel Genellina wrote:
>

>First I want to say that __getitem__ should raise IndexError, not
>KeyError, to indicate "not found"

>
How do you know the Original Poster's class was meant to be a sequence
rather than a mapping?

Ouch, no, sorry, disregard that comment.

--
Gabriel Genellina