Pythonic way for missing dict keys

[Alex Popescu]

Hi all!

I am pretty sure this has been asked a couple of times, but I don't seem
to find it on the archives (Google seems to have a couple of problems
lately).

I am wondering what is the most pythonic way of dealing with missing
keys and default values.

According to my readings one can take the following approaches:

1/ check before (this has a specific name and acronym that I haven't
learnt yet by heart)

if not my_dict.has_key (key):
my_obj = myobject()
my_dict[key] = my_obj
else:
my_obj = my_dict[key]

2/ try and react on error (this has also a specific name, but...)

try:
my_obj = my_dict[key]
except AttributeError:
my_obj = myobject()
my_dict[key] = my_obj

3/ dict.get usage:

my_obj = my_dict.get(key , myobject())

I am wondering which one is the most recommended way? get usage seems
the clearest, but the only problem I see is that I think myobject() is
evaluated at call time, and so if the initialization is expensive you
will probably see surprises.

thanks in advance,
../alex
--
..w( the_mindstorm )p.

[Neil Cerutti]

Re: Pythonic way for missing dict keys

On 2007-07-20, Alex Popescu <the.mindstorm. mailinglist@gma il.comwrote:

Hi all!
>
I am pretty sure this has been asked a couple of times, but I
don't seem to find it on the archives (Google seems to have a
couple of problems lately).
>
I am wondering what is the most pythonic way of dealing with missing
keys and default values.
>
According to my readings one can take the following approaches:

There's also the popular collections.def aultdict.

Usually, the get method of normal dicts is what I want. I use a
defaultdict only when the implicit addition to the dictionary of
defaulted elements is what I really want.

--
Neil Cerutti

[Alex Popescu]

Re: Pythonic way for missing dict keys

Neil Cerutti <horpner@yahoo. comwrote in
news:slrnfa2325 .n4.horpner@FIA D06.norwich.edu :

On 2007-07-20, Alex Popescu <the.mindstorm. mailinglist@gma il.comwrote:

>Hi all!
>>
>I am pretty sure this has been asked a couple of times, but I
>don't seem to find it on the archives (Google seems to have a
>couple of problems lately).
>>
>I am wondering what is the most pythonic way of dealing with missing
>keys and default values.
>>
>According to my readings one can take the following approaches:

>
There's also the popular collections.def aultdict.
>
Usually, the get method of normal dicts is what I want. I use a
defaultdict only when the implicit addition to the dictionary of
defaulted elements is what I really want.
>

This looks like the closest to my needs, but in my case the default value
involves the creation of a custom object instance that is taking parameters
from the current execution context, so I am not very sure I can use it.

../alex
--
..w( the_mindstorm )p.

[Jakub Stolarski]

Re: Pythonic way for missing dict keys

Version 1 and 2 do different thing than version 3. The latter doesn't
add value to dict.

As it was mentioned before, use:
1 - if you expect that there's no key in dict
2 - if you expect that there is key in dict

[Alex Popescu]

Re: Pythonic way for missing dict keys

Jakub Stolarski <jakub.stolarsk i@gmail.comwrot e in
news:1184961448 .456134.153020@ k79g2000hse.goo glegroups.com:

Version 1 and 2 do different thing than version 3. The latter doesn't
add value to dict.
>
As it was mentioned before, use:
1 - if you expect that there's no key in dict
2 - if you expect that there is key in dict
>

I may be missing something but I think the 3 approaches are completely
equivalent in terms of functionality.

../alex
--
..w( the_mindstorm )p.

[Carsten Haese]

Re: Pythonic way for missing dict keys

On Fri, 2007-07-20 at 19:39 +0000, Alex Popescu wrote:

Neil Cerutti <horpner@yahoo. comwrote in
news:slrnfa2325 .n4.horpner@FIA D06.norwich.edu :
>

On 2007-07-20, Alex Popescu <the.mindstorm. mailinglist@gma il.comwrote:

Hi all!
>
I am pretty sure this has been asked a couple of times, but I
don't seem to find it on the archives (Google seems to have a
couple of problems lately).
>
I am wondering what is the most pythonic way of dealing with missing
keys and default values.
>
According to my readings one can take the following approaches:

There's also the popular collections.def aultdict.

Usually, the get method of normal dicts is what I want. I use a
defaultdict only when the implicit addition to the dictionary of
defaulted elements is what I really want.

>
This looks like the closest to my needs, but in my case the default value
involves the creation of a custom object instance that is taking parameters
from the current execution context, so I am not very sure I can use it.

If by "current execution context" you mean globally visible names, this
should still be possible:

>>from collections import defaultdict
>>def make_default():

.... return x+y
....

>>dd = defaultdict(mak e_default)
>>x = 40
>>y = 2
>>print dd[0]

42

>>x = "Dead"
>>y = " Parrot"
>>print dd[1]

Dead Parrot

>>print dd

defaultdict(<fu nction make_default at 0xb7f71e2c>, {0: 42, 1: 'Dead Parrot'})

HTH,

--
Carsten Haese

http://informixdb.sourceforge.net

[Steven D'Aprano]

Re: Pythonic way for missing dict keys

On Fri, 20 Jul 2007 19:08:57 +0000, Alex Popescu wrote:

I am wondering what is the most pythonic way of dealing with missing
keys and default values.

[snip three versions]

Others have already mentioned the collections.def aultdict type, however it
seems people have forgotten about the setdefault method of dictionaries.

value = somedict.setdef ault(key, defaultvalue)

The disadvantage of setdefault is that the defaultvalue has to be created
up front. The disadvantage of collections.def aultdict is that the "default
factory" function takes no arguments, which makes it rather less than
convenient. One can work around this using global variables:

# The default value is expensive to calculate, and known
# only at runtime.

>>expensivefunc tion = lambda x: str(x)
>>D = collections.def aultdict(lambda : expensivefuncti on(context))

# lots of code goes here...

# set context just before fetching from the default dict

>>context = 42
>>value = D['key']
>>print value, D

42 defaultdict(<fu nction <lambdaat 0xb7eb4fb4>, {'key': '42'})

but one should be very leery of relying on global variables like that.

That suggests the best solution is something like this:

def getdefault(adic t, key, expensivefuncti on, context):
if key in adict:
return adict[key]
else:
value = expensivefuncti on(context)
adict[key] = value
return value



--
Steven.

[Rustom Mody]

Re: Pythonic way for missing dict keys

Can someone who knows about python internals throw some light on why

>>x in dic

is cheaper than

>>dic.has_key(x )

??

[Carsten Haese]

Re: Pythonic way for missing dict keys

On Sat, 21 Jul 2007 09:22:32 +0530, Rustom Mody wrote

Can someone who knows about python internals throw some light on why

>x in dic

is cheaper than

>dic.has_key( x)

>
??

I won't claim to know Python internals, but compiling and disassembling the
expressions in question reveals the reason:

>>from compiler import compile
>>from dis import dis
>>dis(compile(" dic.has_key(x)" ,"","eval"))

1 0 LOAD_NAME 0 (dic)
3 LOAD_ATTR 1 (has_key)
6 LOAD_NAME 2 (x)
9 CALL_FUNCTION 1
12 RETURN_VALUE

>>dis(compile(" x in dic","","eval") )

1 0 LOAD_NAME 0 (x)
3 LOAD_NAME 1 (dic)
6 COMPARE_OP 6 (in)
9 RETURN_VALUE

"dic.has_key(x) " goes through an attribute lookup to find the function that
looks for the key. "x in dic" finds the function more directly.

--
Carsten Haese

http://informixdb.sourceforge.net

[Bruno Desthuilliers]

Re: Pythonic way for missing dict keys

Alex Popescu a écrit :

Hi all!
>
I am pretty sure this has been asked a couple of times, but I don't seem
to find it on the archives (Google seems to have a couple of problems
lately).
>
I am wondering what is the most pythonic way of dealing with missing
keys and default values.
>
According to my readings one can take the following approaches:
>
1/ check before (this has a specific name and acronym that I haven't
learnt yet by heart)
>
if not my_dict.has_key (key):
my_obj = myobject()
my_dict[key] = my_obj
else:
my_obj = my_dict[key]

if key not in my_dict:
my_obj = my_dict[key] = myobject()
else:
my_obj = my_dict[key]

2/ try and react on error (this has also a specific name, but...)
>
try:
my_obj = my_dict[key]
except AttributeError:
my_obj = myobject()
my_dict[key] = my_obj

cf above for a shortcut...

3/ dict.get usage:
>
my_obj = my_dict.get(key , myobject())

Note that this last one won't have the same result, since it won't store
my_obj under my_dict[key]. You'd have to use dict.setdefault :

my_obj = my_dict.setdefa ult(key, myobject())

I am wondering which one is the most recommended way?

It depends on the context. wrt/ 1 and 2, use 1 if you expect that most
of the time, my_dict[key] will not be set, and 2 if you expect that most
of the time, my_dict[key] will be set.

get usage seems
the clearest, but the only problem I see is that I think myobject() is
evaluated at call time,

Myobject will be instanciated each time, yes.

and so if the initialization is expensive you
will probably see surprises.

No "surprise" here, but it can indeed be suboptimal if instanciating
myobject is costly.

[Bruno Desthuilliers]

Re: Pythonic way for missing dict keys

Alex Popescu a écrit :

Jakub Stolarski <jakub.stolarsk i@gmail.comwrot e in
news:1184961448 .456134.153020@ k79g2000hse.goo glegroups.com:
>
>

>>Version 1 and 2 do different thing than version 3. The latter doesn't
>>add value to dict.
>>
>>As it was mentioned before, use:
>>1 - if you expect that there's no key in dict
>>2 - if you expect that there is key in dict
>>

>
>
I may be missing something

You are.

but I think the 3 approaches are completely
equivalent in terms of functionality.

d = dict()
answer = d.get('answer', 42)
answer in d
=False

[Alex Martelli]

Re: Pythonic way for missing dict keys

Carsten Haese <carsten@uniqsy s.comwrote:

On Sat, 21 Jul 2007 09:22:32 +0530, Rustom Mody wrote

Can someone who knows about python internals throw some light on why

>>x in dic

is cheaper than

>>dic.has_key(x )

??

>
I won't claim to know Python internals, but compiling and disassembling the
expressions in question reveals the reason:
>

>from compiler import compile
>from dis import dis
>dis(compile("d ic.has_key(x)", "","eval"))

1 0 LOAD_NAME 0 (dic)
3 LOAD_ATTR 1 (has_key)
6 LOAD_NAME 2 (x)
9 CALL_FUNCTION 1
12 RETURN_VALUE

>dis(compile( "x in dic","","eval") )

1 0 LOAD_NAME 0 (x)
3 LOAD_NAME 1 (dic)
6 COMPARE_OP 6 (in)
9 RETURN_VALUE
>
"dic.has_key(x) " goes through an attribute lookup to find the function that
looks for the key. "x in dic" finds the function more directly.

Yup, it's mostly that, as microbenchmarki ng can confirm:

brain:~ alex$ python -mtimeit -s'd={}; f=d.has_key' 'f(23)'
10000000 loops, best of 3: 0.146 usec per loop
brain:~ alex$ python -mtimeit -s'd={}; f=d.has_key' '23 in d'
10000000 loops, best of 3: 0.142 usec per loop
brain:~ alex$ python -mtimeit -s'd={}; f=d.has_key' 'f(23)'
10000000 loops, best of 3: 0.146 usec per loop
brain:~ alex$ python -mtimeit -s'd={}; f=d.has_key' '23 in d'
10000000 loops, best of 3: 0.142 usec per loop
brain:~ alex$ python -mtimeit -s'd={}; f=d.has_key' 'd.has_key(23)'
1000000 loops, best of 3: 0.278 usec per loop
brain:~ alex$ python -mtimeit -s'd={}; f=d.has_key' 'd.has_key(23)'
1000000 loops, best of 3: 0.275 usec per loop

the in operator still appears to have a tiny repeatable advantage (about
4 nanoseconds on my laptop) wrt even the hoisted method, but the
non-hoisted method, due to repeated lookup, is almost twice as slow
(over 100 nanoseconds penalty, on my laptop).


Alex

[Alex Popescu]

Re: Pythonic way for missing dict keys

Bruno Desthuilliers <bdesth.quelque chose@free.quel quepart.frwrote in
news:46a20fda$0 $27858$426a74cc @news.free.fr:

Alex Popescu a écrit :

>Jakub Stolarski <jakub.stolarsk i@gmail.comwrot e in

>

>
[snip...]
>
>
d = dict()
answer = d.get('answer', 42)
answer in d
=False
>

Thanks. I think to make the 3rd approach completely equivalent I should
have been using d.setdefault(ke y, myojbect()).

../alex
--
..w( the_mindstorm )p.

[Duncan Booth]

Re: Pythonic way for missing dict keys

"Rustom Mody" <rustompmody@gm ail.comwrote:

Can someone who knows about python internals throw some light on why

>>>x in dic

is cheaper than

>>>dic.has_key( x)

>
??
>

Some special methods are optimised by having a reserved slot in the data
structure used to implement a class. The 'in' operator uses one of these
slots so it can bypass all the overheads of looking up an attribute such as
'has_key'.

[Zentrader]

Re: Pythonic way for missing dict keys

On Jul 21, 7:48 am, Duncan Booth <duncan.bo...@i nvalid.invalidw rote:

"Rustom Mody" <rustompm...@gm ail.comwrote:

Can someone who knows about python internals throw some light on why

>>x in dic

is cheaper than

>>dic.has_key(x )

>

??

>

>From the 2.6 PEP #361 (looks like dict.has_key is deprecated)

Python 3.0 compatability: ['compatibility'-->someone should use a
spell-checker for 'official' releases]
- warnings were added for the following builtins which no
longer exist in 3.0:
apply, callable, coerce, dict.has_key, execfile, reduce,
reload