A problem with exec statement

[TPJ]

I have the following code:

-----------------------------------
def f():

def g():
a = 'a' # marked line 1
exec 'a = "b"' in globals(), locals()
print "g: a =", a

a = 'A' # marked line 2
exec 'a = "B"' in globals(), locals()
print "f: a =", a
g()

f()
-----------------------------------

I don't understand, why its output is:

f: a = A
g: a = a

instead of:

f: a = B
g: a = b

All works as intended, if the marked lines are commented out. I just
don't understand, why. (I suppose I don't understand, how the exec
statement works, or the way Python handles objects, their names and
namespaces...) In my opinion (according to my knowledge about Python),
with or without the marked lines commented, the code should work the
same. Well - I think I have to learn more about Python...

According to my knowledge, the most important are the namespaces: the
local ones, in this case. When Python calls the f function, its
namespace is created. This namespace contains only the g function.
Then the a variable is created (and the "a" name is added to the f
function namespace).

The next statement is the exec one. Since the statement "knows" the
local namespace (obtained from the locals() function), it should
replace the value of the a variable in the local namespace with the
value of the new string "B". I don't understand, why this is not done.

The situation in the g function is similar, the only difference is
that the local namespace contains the "a" name, that refers to a
different Python object.

[Peter Otten]

Re: A problem with exec statement

TPJ wrote:
[color=blue]
> I have the following code:
>
> -----------------------------------
> def f():
>
>   def g():
>     a = 'a'             # marked line 1
>     exec 'a = "b"' in globals(), locals()
>     print "g: a =", a
>
>   a = 'A'               # marked line 2
>   exec 'a = "B"' in globals(), locals()
>   print "f: a =", a
>   g()
>
> f()
> -----------------------------------
>
> I don't understand, why its output is:
>
> f: a = A
> g: a = a
>
> instead of:
>
> f: a = B
> g: a = b[/color]

Use the exec statement without the in-clause to get the desired effect:
[color=blue][color=green][color=darkred]
>>> def f():[/color][/color][/color]
.... a = "a"
.... exec "a = 'B'"
.... print a
....[color=blue][color=green][color=darkred]
>>> f()[/color][/color][/color]
B

Inside a function locals() creates a new dictionary with name/value pairs of
the variables in the local namespace every time you call it. When that
dictionary is modified the local variables are *not* updated accordingly.
[color=blue][color=green][color=darkred]
>>> def f():[/color][/color][/color]
.... a = "a"
.... d = locals()
.... exec "a = 'B'" in globals(), d
.... print a, d["a"]
....[color=blue][color=green][color=darkred]
>>> f()[/color][/color][/color]
a B

By the way, experiments on the module level are likely to confuse because
there locals() returns the same dictionary as globals().

Peter

[Steve Holden]

Re: A problem with exec statement

TPJ wrote:[color=blue]
> I have the following code:
>
> -----------------------------------
> def f():
>
> def g():
> a = 'a' # marked line 1
> exec 'a = "b"' in globals(), locals()
> print "g: a =", a
>
> a = 'A' # marked line 2
> exec 'a = "B"' in globals(), locals()
> print "f: a =", a
> g()
>
> f()
> -----------------------------------
>
> I don't understand, why its output is:
>
> f: a = A
> g: a = a
>
> instead of:
>
> f: a = B
> g: a = b
>
> All works as intended, if the marked lines are commented out. I just
> don't understand, why. (I suppose I don't understand, how the exec
> statement works, or the way Python handles objects, their names and
> namespaces...) In my opinion (according to my knowledge about Python),
> with or without the marked lines commented, the code should work the
> same. Well - I think I have to learn more about Python...
>[/color]
That's true of almost everybody reading this list, myself included.
[color=blue]
> According to my knowledge, the most important are the namespaces: the
> local ones, in this case. When Python calls the f function, its
> namespace is created. This namespace contains only the g function.
> Then the a variable is created (and the "a" name is added to the f
> function namespace).
>[/color]
That's a pretty good summary. In fact just after the call to f is
started its namespace doesn't even contain "g", that's added by
executing the def statement that defines g.

The assignment does indeed create the name "a" in the function's (local)
namespace.
[color=blue]
> The next statement is the exec one. Since the statement "knows" the
> local namespace (obtained from the locals() function), it should
> replace the value of the a variable in the local namespace with the
> value of the new string "B". I don't understand, why this is not done.
>[/color]
So when you exec 'a = "B"' in globals(), locals() you might think you
were changing the local namespace. In fact you are changing a *copy* of
the local namespace: if you read the documentation carefully you will
see under locals() it says """Warning: The contents of this dictionary
should not be modified; changes may not affect the values of local
variables used by the interpreter."""
[color=blue]
> The situation in the g function is similar, the only difference is
> that the local namespace contains the "a" name, that refers to a
> different Python object.
>[/color]
The same answer presumably pertains here. If you modify your code to read:

def f():

def g():
a = 'a' # marked line 1
print "globals:", globals(), '\nlocals:', locals()
exec 'a = "b"' in globals(), locals()
print "globals:", globals(), '\nlocals:', locals()
print "g: a =", a

a = 'A' # marked line 2
print "Globals:", globals(), '\nLocals:', locals()
exec 'a = "B"' in globals(), locals()
print "Globals:", globals(), '\nLocals:', locals()
print "f: a =", a
g()

f()

you will see quite clearly that you aren't making the changes you
anticipate to the local namespace. I hope I have explained why.

One last note. Newcomers to Python often seem fascinated by the ability
to use exec to achieve namespace indirection. Even allowing for the
difficulties you've already experienced, it's nearly always better in
practical cases to use assignment to the keys of a dictionary. Then no
exec is required, and you have direct control over your own namespace.

regards
Steve

--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd www.holdenweb.com
Love me, love my blog holdenweb.blogs pot.com

[TPJ]

Re: A problem with exec statement

> Use the exec statement without the in-clause to get the desired effect:[color=blue]
>[color=green][color=darkred]
> >>> def f():[/color][/color]
> ... a = "a"
> ... exec "a = 'B'"
> ... print a
> ...[color=green][color=darkred]
> >>> f()[/color][/color]
> B
> <snip>[/color]

Well... I *do* realize that. But this is *not* my problem. I have a
function with another nested one. If I used "exec ..." instead of "exec
.... in some_dict, some_dict" I would get the "SyntaxErro r: unqualified
exec is not allowed in function 'f' it contains a nested function with
free variables".

To be honest, the message cited above is the answer to the question
"Why have I put those globals(), locals() in the exec statments?".

[TPJ]

Re: A problem with exec statement

> <snip>[color=blue]
> So when you exec 'a = "B"' in globals(), locals() you might think you
> were changing the local namespace. In fact you are changing a copy of
> the local namespace:
> <snip>[/color]

Well, that explains much, but not all that I want to be explained. Why?
Because now I understand, that by invoking

exec "a = 'B'" in globals(), locals()

I can modify only *copies* of the global and local namespaces dicts,
not the dicts themselves. OK, that's why my code doesn't work as I want
it to work.

But why on Earth *the same code* will work, if I remove the assignments
from the marked lines? Why then operating on copies of the local
namespaces dicts *will work* ?
[color=blue]
> (...) Even allowing for the
> difficulties you've already experienced, it's nearly always better in
> practical cases to use assignment to the keys of a dictionary. Then no
> exec is required, and you have direct control over your own namespace.[/color]

Well... Is this a sugestion, that instead of messing up with the exec
statements used to modify local namespaces I should use dictionaries?

Perhaps you're right. In fact, the problem, that I'm trying to solve is
as follows:

def funcA():

def funcB():
...
var1, var2, var3, ..., varN = ( None, ) * N
t = ( (regexp1, 'var1'), (regexp2, 'var2'), ..., (regexpN, 'varN')
)
for regexp, var_name in t:
match = regexp.match( some_string )
if match != None:
# now the *magic* exec statement comes...
exec var_name + ' = match.groups()[0]' in globals(), locals()
return var1, var2, var3, ..., varN

...
k1, k2, k3, ..., kN = funcB()

Of course, the code presented above doesn't work. It works, if one
change is done in the function funcB:

def funcB():
...
# no creation of any local variables called var1, var2, ..., varN
here
t = ( (regexp1, 'var1'), (regexp2, 'var2'), ..., (regexpN, 'varN')
)
for regexp, var_name in t:
match = regexp.match( some_string )
if match != None:
# now the *magic* exec statement comes...
exec var_name + ' = match.groups()[0]' in globals(), locals()
else:
# here we put the code, that will assign None to the variable
exec var_name + ' = None'
return var1, var2, var3, ..., varN

But I *still* don't understand, why this code works, if I operate on a
copy of the local namespace dict...

Of course, I can do the same thing in a different matter - by using a
dictionary. And perhaps I will. But I still want to know, how the exec
statement works.

* * *

My problem is more complicated, that the presented example. In general,
my problem is: how to create a local variable by executing the Python
code, that isn't known at the moment of writing the program? In another
words: I have to create a local variable, whose name will be known at
the runtime, in a nested function.

Is it possible, or have I to use dictionaries, instead of exec
statement used to modify local namespaces?

[Peter Otten]

Re: A problem with exec statement

TPJ wrote:
[color=blue][color=green]
>> (...) Even allowing for the
>> difficulties you've already experienced, it's nearly always better in
>> practical cases to use assignment to the keys of a dictionary. Then no
>> exec is required, and you have direct control over your own namespace.[/color]
>
> Well... Is this a sugestion, that instead of messing up with the exec
> statements used to modify local namespaces I should use dictionaries?
>
> Perhaps you're right. In fact, the problem, that I'm trying to solve is
> as follows:
>
> def funcA():
>
> def funcB():
> ...
> var1, var2, var3, ..., varN = ( None, ) * N
> t = ( (regexp1, 'var1'), (regexp2, 'var2'), ..., (regexpN, 'varN')
> )
> for regexp, var_name in t:
> match = regexp.match( some_string )
> if match != None:
> # now the *magic* exec statement comes...
> exec var_name + ' = match.groups()[0]' in globals(), locals()
> return var1, var2, var3, ..., varN
>
> ...
> k1, k2, k3, ..., kN = funcB()[/color]
[color=blue]
> My problem is more complicated, that the presented example. In general,
> my problem is: how to create a local variable by executing the Python
> code, that isn't known at the moment of writing the program? In another
> words: I have to create a local variable, whose name will be known at
> the runtime, in a nested function.
>
> Is it possible, or have I to use dictionaries, instead of exec
> statement used to modify local namespaces?[/color]

There is a mismatch between your example code and the problem description
you are giving. The example can easily be rewritten without nested scopes
and exec:

# of course untested
def funcB(some_stri ng):
for r in [regexp1, regexp2, regexp3, ..., regexpN]:
match = r.match(some_st ring)
if match:
yield match.group(1)
else:
yield None

def funcA():
k1, k2, k3, ..., kN = funcB(some_stri ng)

The uniform ki variable names are an indication that you may be able to
simplify this even further. I'm therefore confident that rewriting your
real code without exec will be more rewarding than digging deeper into the
quirks of exec (which admittedly I find puzzling, too).

Peter

PS: Here is another gem showing that my original assertion that inside a
function locals() is always a copy is wrong:
[color=blue][color=green][color=darkred]
>>> def f():[/color][/color][/color]
.... locals()["a"] = 42
.... print a
....[color=blue][color=green][color=darkred]
>>> f()[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in f
NameError: global name 'a' is not defined[color=blue][color=green][color=darkred]
>>> def f():[/color][/color][/color]
.... locals()["a"] = 42
.... print a
.... if False: exec ""
....[color=blue][color=green][color=darkred]
>>> f()[/color][/color][/color]
42