closures and dynamic binding

[Aaron \Castironpi\ Brady]

Hello all,

To me, this is a somewhat unintuitive behavior. I want to discuss the
parts of it I don't understand.

>>f= [ None ]* 10
>>for n in range( 10 ):

.... f[ n ]= lambda: n
....

>>f[0]()

9

>>f[1]()

9

I guess I can accept this part so far, though it took a little getting
used to. I'm writing some code and found the following workaround,
but I don't think it should give different results. Maybe I'm not
understanding some of the details of closures.

>>f= [ None ]* 10
>>for n in range( 10 ):

.... f[ n ]= (lambda n: ( lambda: n ) )( n )
....

>>f[0]()

0

>>f[1]()

1

Which is of course the desired effect. Why doesn't the second one
just look up what 'n' is when I call f[0], and return 9?

[Marc 'BlackJack' Rintsch]

Re: closures and dynamic binding

On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi \" Brady wrote:

To me, this is a somewhat unintuitive behavior. I want to discuss the
parts of it I don't understand.
>

>>>f= [ None ]* 10
>>>for n in range( 10 ):

... f[ n ]= lambda: n
...

>>>f[0]()

9

>>>f[1]()

9

`n` is looked up at the time ``f[0]`` is called. At that time it is
bound to 9.

>>>f= [ None ]* 10
>>>for n in range( 10 ):

... f[ n ]= (lambda n: ( lambda: n ) )( n ) ...

>>>f[0]()

0

>>>f[1]()

1
>
Which is of course the desired effect. Why doesn't the second one just
look up what 'n' is when I call f[0], and return 9?

It *does* look up `n` at the time you call ``f[0]`` but this time it's
the local `n` of the outer ``lambda`` function and that is bound to
whatever the function was called with. At the time it was called the
global `n` was bound to 0. Maybe it get's more clear if you don't name
it `n`:

In [167]: f = [None] * 10

In [168]: for n in xrange(10):
.....: f[n] = (lambda x: lambda: x)(n)
.....:

In [169]: f[0]()
Out[169]: 0

Ciao,
Marc 'BlackJack' Rintsch

[Aaron \Castironpi\ Brady]

Re: closures and dynamic binding

On Sep 28, 1:14 am, Marc 'BlackJack' Rintsch <bj_...@gmx.net wrote:

On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi \" Brady wrote:

To me, this is a somewhat unintuitive behavior.  I want to discuss the
parts of it I don't understand.

>

>>f= [ None ]* 10
>>for n in range( 10 ):

...     f[ n ]= lambda: n
...

>>f[0]()

9

>>f[1]()

9

>
`n` is looked up at the time ``f[0]`` is called.  At that time it is
bound to 9.
>

>>f= [ None ]* 10
>>for n in range( 10 ):

...     f[ n ]= (lambda n: ( lambda: n ) )( n ) ...

>>f[0]()

0

>>f[1]()

1

>

Which is of course the desired effect.  Why doesn't the second one just
look up what 'n' is when I call f[0], and return 9?

>
It *does* look up `n` at the time you call ``f[0]`` but this time it's
the local `n` of the outer ``lambda`` function and that is bound to
whatever the function was called with.  At the time it was called the
global `n` was bound to 0.  Maybe it get's more clear if you don't name
it `n`:
>
In [167]: f = [None] * 10
>
In [168]: for n in xrange(10):
   .....:     f[n] = (lambda x: lambda: x)(n)
   .....:
>
In [169]: f[0]()
Out[169]: 0
>
Ciao,
        Marc 'BlackJack' Rintsch

Hi Marc,

It's my understanding that 'n' gets a new value every time through the
loop. n= 0 on the first pass, n= 1 on the second pass, and so on. n=
9 by the end, and that's why `lambda: n` always returns 9. It queries
the variable 'n', and finds 9. (This got lengthy. I started thinking
aloud.)

In your version of the indirect example, it queries the variable 'x',
which it finds in a new distinct scope in each f element. In f[0], x=
0. In f[1], x= 1. There are 10 different 'x' variables throughout
the contents of f. The direct example does not do this allocation of
ten different 'x's.

It's sort of helping. I think I feel like the following is more like
what I'm looking for:

(Non-standard)

>>f= [ None ]* 10
>>for n in range( 10 ):

.... f[ n ]= n
....

>>f[0]

9

>>f[1]

9

because when you access f[0], it looks up the variable 'n'. Obviously
not.

(Unproduced)

>>f= [ None ]* 10
>>for n in range( 10 ):

.... f[ n ]= late( n ) #late/lazy
....

>>f[0]

9

>>f[1]

9

>>f= [ None ]* 10
>>for n in range( 10 ):

.... f[ n ]= early( n ) #early/eager
....

>>f[0]

0

>>f[1]

1

For the functions, I want a function that returns 'n', either late or
early.

(Unproduced)

>>for n in range( 10 ):

.... f[ n ]= lambda: late( n )

>>f[0]()

9

>>for n in range( 10 ):

.... f[ n ]= lambda: early( n )

>>f[0]()

0

I don't think you could pull this off. 'late' and 'early' succeed
with quotes around n, early('n') and late('n'), in the direct
assignments, but the functions aren't so lucky. 'n' has gone on to a
better world by the time 'early' gets any control.

This might have some success.

(Unproduced)

>>for n in range( 10 ):

.... f[ n ]= late( lambda: n )

>>f[0]()

9

>>for n in range( 10 ):

.... f[ n ]= early( lambda: n )

>>f[0]()

0

Though it's beyond my foresight to tell if it's feasible as stated, if
you need quotes, how much introspection you would need, etc. Plus,
'late' and 'early' were accepting quoted parameters earlier. How
would they look inside a function? Could a simple decorator provide
the service?

On a tangent about mutables, it's not that numbers, strings, and
tuples are 'immutable' per se, it's just that they don't have any
methods which mutate them (or assignable properties). Lists and
dictionaries do. It's up to the user whether a custom class does.

[Terry Reedy]

Re: closures and dynamic binding

Aaron "Castironpi " Brady wrote:

Hello all,
>
To me, this is a somewhat unintuitive behavior. I want to discuss the
parts of it I don't understand.
>

>>>f= [ None ]* 10
>>>for n in range( 10 ):

... f[ n ]= lambda: n

This is equivalent to

for n in range(10):
def g(): return n
f[n] = g

which is equivalent to

def g(): return n
f = [g]*10
n = 9

>>>f[0]()

9

>>>f[1]()

9

which make this not so surprising as the original lambda version is to
some people.

I guess I can accept this part so far, though it took a little getting
used to. I'm writing some code and found the following workaround,
but I don't think it should give different results. Maybe I'm not
understanding some of the details of closures.
>

>>>f= [ None ]* 10
>>>for n in range( 10 ):

... f[ n ]= (lambda n: ( lambda: n ) )( n )

This is equivalent to

for n in range(10):
def g(n):
def h:
return n
return h
f[n] = g(n)

Now, to avoid the needless confusion of 'n's, g is equivalent to

def g(x):
def h:
return x
return h

(One could do the same change in the lambdas, too, of course).
so that g(n)() == n, with n stored in each closure h...

...

>>>f[0]()

0

>>>f[1]()

1

to be regurgitated when each is called.

Terry Jan Reedy

[Steven D'Aprano]

Re: closures and dynamic binding

On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi \" Brady wrote:

Hello all,
>
To me, this is a somewhat unintuitive behavior. I want to discuss the
parts of it I don't understand.
>

>>>f= [ None ]* 10
>>>for n in range( 10 ):

... f[ n ]= lambda: n
...

>>>f[0]()

9

>>>f[1]()

9
>
I guess I can accept this part so far, though it took a little getting
used to. I'm writing some code and found the following workaround, but
I don't think it should give different results. Maybe I'm not
understanding some of the details of closures.
>

>>>f= [ None ]* 10
>>>for n in range( 10 ):

... f[ n ]= (lambda n: ( lambda: n ) )( n )
...

>>>f[0]()

0

>>>f[1]()

1
>
Which is of course the desired effect. Why doesn't the second one just
look up what 'n' is when I call f[0], and return 9?

That's an awfully complicated solution. A much easier way to get the
result you are after is to give each function its own local copy of n:

f[n] = lambda n=n: n

As for why the complicated version works, it may be clearer if you expand
it from a one-liner:

# expand: f[ n ]= (lambda n: ( lambda: n ) )( n )

inner = lambda: n
outer = lambda n: inner
f[n] = outer(n)

outer(0) =inner with a local scope of n=0
outer(1) =inner with a local scope of n=1 etc.

Then, later, when you call inner() it grabs the local scope and returns
the number you expected.


--
Steven

[Aaron \Castironpi\ Brady]

Re: closures and dynamic binding

On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com .auwrote:

On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi \" Brady wrote:

Hello all,

>

To me, this is a somewhat unintuitive behavior.  I want to discuss the
parts of it I don't understand.

>

>>f= [ None ]* 10
>>for n in range( 10 ):

...     f[ n ]= lambda: n
...

>>f[0]()

9

>>f[1]()

9

>

I guess I can accept this part so far, though it took a little getting
used to.  I'm writing some code and found the following workaround, but
I don't think it should give different results.  Maybe I'm not
understanding some of the details of closures.

>

>>f= [ None ]* 10
>>for n in range( 10 ):

...     f[ n ]= (lambda n: ( lambda: n ) )( n )
...

>>f[0]()

0

>>f[1]()

1

>

Which is of course the desired effect.  Why doesn't the second one just
look up what 'n' is when I call f[0], and return 9?

>
That's an awfully complicated solution. A much easier way to get the
result you are after is to give each function its own local copy of n:
>
f[n] = lambda n=n: n
>
As for why the complicated version works, it may be clearer if you expand
it from a one-liner:
>
# expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
>
inner = lambda: n
outer = lambda n: inner
f[n] = outer(n)
>
outer(0) =inner with a local scope of n=0
outer(1) =inner with a local scope of n=1 etc.
>
Then, later, when you call inner() it grabs the local scope and returns
the number you expected.
>
--
Steven

Steven,

I must have misunderstood. Here's my run of your code:

>>inner = lambda: n
>>outer = lambda n: inner
>>outer(0)

<function <lambdaat 0x00A01170>

>>a=outer(0)
>>b=outer(1)
>>a()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <lambda>
NameError: global name 'n' is not defined

Why doesn't 'inner' know it's been used in two different scopes, and
look up 'n' based on the one it's in?

[Terry Reedy]

Re: closures and dynamic binding

Aaron "Castironpi " Brady wrote:

On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-

>As for why the complicated version works, it may be clearer if you expand
>it from a one-liner:
>>
># expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
>>
>inner = lambda: n
>outer = lambda n: inner
>f[n] = outer(n)
>>
>outer(0) =inner with a local scope of n=0
>outer(1) =inner with a local scope of n=1 etc.

For this to work, the 'expansion' has to be mental and not actual.
Which is to say, inner must be a text macro to be substituted back into
outer.

>Then, later, when you call inner() it grabs the local scope and returns
>the number you expected.

>
I must have misunderstood. Here's my run of your code:

I cannot speak to what Steven meant, but

>>>inner = lambda: n

when inner is actually compiled outside of outer, it is no longer a
closure over outer's 'n' and 'n' will be looked for in globals instead.

>>>outer = lambda n: inner
>>>outer(0)

<function <lambdaat 0x00A01170>

>>>a=outer(0)
>>>b=outer(1)
>>>a()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <lambda>
NameError: global name 'n' is not defined
>
Why doesn't 'inner' know it's been used in two different scopes, and
look up 'n' based on the one it's in?

That would be dynamic rather than lexical scoping.

[Steven D'Aprano]

Re: closures and dynamic binding

On Sun, 28 Sep 2008 17:47:44 -0400, Terry Reedy wrote:

Aaron "Castironpi " Brady wrote:

>On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-

>

>>As for why the complicated version works, it may be clearer if you
>>expand it from a one-liner:
>>>
>># expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
>>>
>>inner = lambda: n
>>outer = lambda n: inner
>>f[n] = outer(n)
>>>
>>outer(0) =inner with a local scope of n=0 outer(1) =inner with a
>>local scope of n=1 etc.

>
For this to work, the 'expansion' has to be mental and not actual. Which
is to say, inner must be a text macro to be substituted back into outer.

Er, yes, that's what I meant, sorry for not being more explicit. That's
why it wasn't a copy and paste of actual running code.

Or perhaps I just confused myself and was talking nonsense.



--
Steven

[Aaron \Castironpi\ Brady]

Re: closures and dynamic binding

On Sep 28, 4:47 pm, Terry Reedy <tjre...@udel.e duwrote:

Aaron "Castironpi " Brady wrote:

On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-

As for why the complicated version works, it may be clearer if you expand
it from a one-liner:

>

# expand: f[ n ]= (lambda n: ( lambda: n ) )( n )

>

inner = lambda: n
outer = lambda n: inner
f[n] = outer(n)

>

outer(0) =inner with a local scope of n=0
outer(1) =inner with a local scope of n=1 etc.

>
For this to work, the 'expansion' has to be mental and not actual.
Which is to say, inner must be a text macro to be substituted back into
outer.
>

Then, later, when you call inner() it grabs the local scope and returns
the number you expected.

>

I must have misunderstood.  Here's my run of your code:

>
I cannot speak to what Steven meant, but
>

>>inner = lambda: n

>
when inner is actually compiled outside of outer, it is no longer a
closure over outer's 'n' and 'n' will be looked for in globals instead.
>

>>outer = lambda n: inner
>>outer(0)

<function <lambdaat 0x00A01170>

>>a=outer(0)
>>b=outer(1)
>>a()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <lambda>
NameError: global name 'n' is not defined

>

Why doesn't 'inner' know it's been used in two different scopes, and
look up 'n' based on the one it's in?

>
That would be dynamic rather than lexical scoping.

I couldn't find how those apply on the wikipedia website. It says:
"dynamic scoping can be dangerous and almost no modern languages use
it", but it sounded like that was what closures use. Or maybe it was
what 'inner' in Steven's example would use. I'm confused.

Actually, I'll pick this apart a little bit. See above when I
suggested 'late' and 'early' functions which control (or simulate)
different bindings. I get the idea that 'late' bound functions would
use a dangerous "dynamic scope", but I could be wrong; that's just my
impression.

inner = lambda: n
outer = lambda n: inner
f[n] = outer(n)

>

outer(0) =inner with a local scope of n=0
outer(1) =inner with a local scope of n=1 etc.

If you defined these as:

inner= late( lambda: n )
outer= lambda n: inner

You could get the right results. It's not even clear you need
quotes. Perhaps 'late' could carry the definition of 'n' with it when
it's returned from 'outer'.

In my proposal, it makes a copy of the "localest" namespace, at least
all the variables used below it, then returns its argument in an
original closure.

[Terry Reedy]

Re: closures and dynamic binding

Aaron "Castironpi " Brady wrote:

On Sep 28, 4:47 pm, Terry Reedy <tjre...@udel.e duwrote:

>Aaron "Castironpi " Brady wrote:

>>>>>inner = lambda: n

>when inner is actually compiled outside of outer, it is no longer a
>closure over outer's 'n' and 'n' will be looked for in globals instead.
>>

>>>>>outer = lambda n: inner
>>>>>outer(0)
>><function <lambdaat 0x00A01170>
>>>>>a=outer( 0)
>>>>>b=outer( 1)
>>>>>a()
>>Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File "<stdin>", line 1, in <lambda>
>>NameError: global name 'n' is not defined
>>Why doesn't 'inner' know it's been used in two different scopes, and
>>look up 'n' based on the one it's in?

>That would be dynamic rather than lexical scoping.

>
I couldn't find how those apply on the wikipedia website. It says:
"dynamic scoping can be dangerous and almost no modern languages use
it", but it sounded like that was what closures use. Or maybe it was
what 'inner' in Steven's example would use. I'm confused.

As I understand it, partly from postings here years ago...

Lexical: The namespace scope of 'n' in inner is determined by where
inner is located in the code -- where is is compiled. This is Python
(and nearly all modern languages). Even without closures, the global
scope of a function is the module it is defined in.

Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
determined by where inner is called from. This is what you seemed to be
suggesting -- look up 'n' based on the scope it is *used* in.

Even without closures, dynamic scoping would be if the global scope of a
function for each call were the module it is called in.

tjr

[Lawrence D'Oliveiro]

Re: closures and dynamic binding

In message
<3c75ce4b-bb58-4da4-9b7a-87d7c5d5d2d0@m4 5g2000hsb.googl egroups.com>,
Aaron "Castironpi " Brady wrote:

[Wikipedia] says:
"dynamic scoping can be dangerous and almost no modern languages use
it", but it sounded like that was what closures use.

Closures will use whatever the language says they use. LISP used dynamic
binding, but this caused some interesting problems as mentioned above. Perl
allows both, depending on whether the local is declared with "local"
or "my". Python uses only lexical, though there's probably some way to get
dynamic behaviour if you want. :)

[Paul Boddie]

Re: closures and dynamic binding

On 29 Sep, 05:56, Terry Reedy <tjre...@udel.e duwrote:

>
As I understand it, partly from postings here years ago...
>
Lexical: The namespace scope of 'n' in inner is determined by where
inner is located in the code -- where is is compiled.  This is Python
(and nearly all modern languages).  Even without closures, the global
scope of a function is the module it is defined in.

This is how I understand it, too. The confusing part involves the
definition of any inner function and how any "external" names are
bound or defined. As we've seen...

def f(x):
def g():
return x
x += 1 # added for illustration
return g

....it might look at first glance like the function known as g (within
f) should return the initial value of x (as known within f), since
that was the value x had when g was defined. The following is an
example execution trace based on that mental model:

fn = f(1)
-def f(1):
- def g(): # g defined with x as 1
- return x # would be 1
- x += 1 # x becomes 2
- return g
fn()
-def g():
- return x # would still be 1

However, as we know, this isn't the case in real Python since g isn't
initialised with the value of x at the time of its definition - it
instead maintains access to the namespace providing x. We must
therefore revise the example:

fn = f(1)
-def f(1):
- def g(): # g refers to x within f(1)
- return x # would be the current value of x within f(1)
- x += 1 # x becomes 2
- return g
fn()
-def g(): # g refers to x within f(1)
- return x # would be the current value of x within f(1), which is
2

This is the dynamic aspect of closures: values aren't used to
initialise inner functions; names are looked up from their origin.

Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
determined by where inner is called from. This is what you seemed to be
suggesting -- look up 'n' based on the scope it is *used* in.

Indeed. Dynamic scoping is confusing in that one has to set up an
appropriate "environmen t" for the closure to access so that references
to names can be meaningfully satisfied; obviously, this creates all
sorts of encapsulation problems. The confusing aspect of lexical
scoping, however, especially if non-local names can be changed, is
that the effects of closures are potentially distant - one doesn't
always invoke inner functions in the place where they were defined, as
we see above - and such effects may thus happen within "abandoned
namespaces" - that is, namespaces which can no longer be revisited and
used in their original context. So, in the above, once f has been
invoked, the namespace for that invocation effectively lives on, but
that namespace is not a general one for f - if we invoke f again, we
get another namespace as one should reasonably expect.

A somewhat separate issue is illustrated by the modification of x
within f. Although for most statements, we would expect the value of x
to evolve following from a top-to-bottom traversal of the code within
a unit, function definition statements do not behave like, say, "for",
"if" or "while" statements. Now although this should be obvious at the
module global level, I feel that it can be easy to overlook within a
function where one normally expects to find plain old control-flow
constructs, expressions, assignments and so on. It is pertinent to
note, with respect to the original inquiry, that lambda functions are
subject to the same caveats, and I believe that lexical scoping was
introduced precisely to make lambda functions less cumbersome to
employ, eliminating the need to explicitly initialise them using the
"identity" default parameters trick, but obviously introducing the
consequences and potential misunderstandin gs described above.

Paul

[Michele Simionato]

Re: closures and dynamic binding

On Sep 28, 6:43 am, "Aaron \"Castironpi \" Brady"
<castiro...@gma il.comwrote:

Hello all,
>
To me, this is a somewhat unintuitive behavior.  I want to discuss the
parts of it I don't understand.
>

>f= [ None ]* 10
>for n in range( 10 ):

>
...     f[ n ]= lambda: n
...>>f[0]()
9

>f[1]()

>
9
>

You may want to read this old thread that goes in detail explaining
scoping issues
with closures in for loops:

http://groups.google.com/group/comp.lang.python/browse_frm/thread/d691240a5cfebcdf/93503c5b9c66226e?hl=en&lnk=gst&q=simionato+jacek#93503c5b9c66226e

[Terry Reedy]

Re: closures and dynamic binding

Paul Boddie wrote:

On 29 Sep, 05:56, Terry Reedy <tjre...@udel.e duwrote:

....

>Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
>determined by where inner is called from. This is what you seemed to be
>suggesting -- look up 'n' based on the scope it is *used* in.

....

A somewhat separate issue is illustrated by the modification of x
within f. Although for most statements, we would expect the value of x
to evolve following from a top-to-bottom traversal of the code within
a unit, function definition statements do not behave like, say, "for",
"if" or "while" statements. Now although this should be obvious at the
module global level, I feel that it can be easy to overlook within a
function where one normally expects to find plain old control-flow
constructs, expressions, assignments and so on. It is pertinent to
note, with respect to the original inquiry, that lambda functions are
subject to the same caveats,

Please: Python does not have 'lambda functions'. Def statements and
lambda expressions both define instances of the function class. So this
amounts to saying "functions are subject to the same caveats as functions."

and I believe that lexical scoping was introduced precisely to make
lambda functions less cumbersome to
employ, eliminating the need to explicitly initialise them using the
"identity" default parameters trick, but obviously introducing the
consequences and potential misunderstandin gs described above.

As I meant when I wrote "Even without closures, the global scope of a
function is the module it is defined in.", the concept of lexical
scoping applies even to non-nested top-level functions.

Yes, closures for nested functions were introduced in part to replace
the pseudo-default parameter trick (which does not require the confusion
of using the same name in both local namespaces). But this applies to
both syntactical forms of function definition and had nothing to do in
particular with easing the quite limited lambda form, which Guido does
not like and once thought to delete in 3.0.

Closures were also introduced because default parameters are limited to
capturing a value at the time of function definition instead of at
function invocation. Consider your example which I have moved down to here.

This is how I understand it, too. The confusing part involves the
definition of any inner function and how any "external" names are
bound or defined. As we've seen...
>
def f(x):
def g():
return x
x += 1 # added for illustration
return g
>
...it might look at first glance like the function known as g (within
f) should return the initial value of x (as known within f), since
that was the value x had when g was defined.

Only if one is confused about the difference between default parameter
expressions, which are evaluated when the function is defined, and body
expressions, which are evaluated when the function is called. If you
want g to use the 'initial' value that x has when g is defined, then say so.

def f(x):
def g(y=x) # or x=x if you really prefer
return y # or x, with the consequent possibility of confusion
x += 1
return g

If 'x' is fixed, this is a difference without effect. If 'x' is not, as
in this pair of examples, the difference is important.

The introduction of the 'nonlocal' keyword will make the latter case
more common and hence the second reason more important. Consider code like

def f(*args)
x = [0]
def g()
...
x[0] += 1
...
<code calling g some variable number of times>
return x[0], ...

This has x fixedly bound to a particular list. The default argument
trick would have worked here. In the future, this can and will be
written more naturally as

def f(*args)
x = 0
def g()
nonlocal x
...
x += 1
...
<code calling g some variable number of times>
return x, ...

With x being rebound, the default argument trick would *not* work. And,
of course, lambda expressions are not in the picture.

Terry Jan Reedy

[Aaron \Castironpi\ Brady]

Re: closures and dynamic binding

On Sep 29, 9:14 am, Paul Boddie <p...@boddie.or g.ukwrote:

On 29 Sep, 05:56, Terry Reedy <tjre...@udel.e duwrote:
>
>
>

As I understand it, partly from postings here years ago...

>

Lexical: The namespace scope of 'n' in inner is determined by where
inner is located in the code -- where is is compiled.  This is Python
(and nearly all modern languages).  Even without closures, the global
scope of a function is the module it is defined in.

>
This is how I understand it, too. The confusing part involves the
definition of any inner function and how any "external" names are
bound or defined. As we've seen...
>
  def f(x):
    def g():
      return x
    x += 1 # added for illustration
    return g
>
...it might look at first glance like the function known as g (within
f) should return the initial value of x (as known within f), since
that was the value x had when g was defined. The following is an
example execution trace based on that mental model:
>
fn = f(1)
-def f(1):
-  def g(): # g defined with x as 1
-    return x # would be 1
-  x += 1 # x becomes 2
-  return g
fn()
-def g():
-  return x # would still be 1
>
However, as we know, this isn't the case in real Python since g isn't
initialised with the value of x at the time of its definition - it
instead maintains access to the namespace providing x. We must
therefore revise the example:
>
fn = f(1)
-def f(1):
-  def g(): # g refers to x within f(1)
-    return x # would be the current value of x within f(1)
-  x += 1 # x becomes 2
-  return g
fn()
-def g(): # g refers to x within f(1)
-  return x # would be the current value of x within f(1), which is
2
>
This is the dynamic aspect of closures: values aren't used to
initialise inner functions; names are looked up from their origin.
>

Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
determined by where inner is called from. This is what you seemed to be
suggesting -- look up 'n' based on the scope it is *used* in.

>
Indeed. Dynamic scoping is confusing in that one has to set up an
appropriate "environmen t" for the closure to access so that references
to names can be meaningfully satisfied; obviously, this creates all
sorts of encapsulation problems. The confusing aspect of lexical
scoping, however, especially if non-local names can be changed, is
that the effects of closures are potentially distant - one doesn't
always invoke inner functions in the place where they were defined, as
we see above - and such effects may thus happen within "abandoned
namespaces" - that is, namespaces which can no longer be revisited and
used in their original context. So, in the above, once f has been
invoked, the namespace for that invocation effectively lives on, but
that namespace is not a general one for f - if we invoke f again, we
get another namespace as one should reasonably expect.
>
A somewhat separate issue is illustrated by the modification of x
within f. Although for most statements, we would expect the value of x
to evolve following from a top-to-bottom traversal of the code within
a unit, function definition statements do not behave like, say, "for",
"if" or "while" statements. Now although this should be obvious at the
module global level, I feel that it can be easy to overlook within a
function where one normally expects to find plain old control-flow
constructs, expressions, assignments and so on. It is pertinent to
note, with respect to the original inquiry, that lambda functions are
subject to the same caveats, and I believe that lexical scoping was
introduced precisely to make lambda functions less cumbersome to
employ, eliminating the need to explicitly initialise them using the
"identity" default parameters trick, but obviously introducing the
consequences and potential misunderstandin gs described above.
>
Paul

I'm thinking of a comparison to try to see an example.

I tried this which still surprised me at first.

>>g= [0]
>>f= lambda: g
>>del g
>>f()

NameError: global name 'g' is not defined

It took a little thinking. The namespace in 'f' is the global
namespace. They are one in the same dictionary. id( f.namespace ) ==
id( main.namespace ). f.namespace is main.namespace.

When f gets a parameter, its namespace is different by one (or two
when two parameters, etc).

>>g=[0]
>>f= lambda h: lambda: h
>>i= f( g )
>>del g
>>i()

[0]

The statement 'del g' removes 'g' from the module/global namespace,
but it lives on in the namespace of 'f', or more accurately, 'i'.

>>g=[0]
>>f= lambda h: lambda: h
>>i= f( g )
>>j= f( g )
>>del g
>>i().append( 1)
>>j()

[0, 1]

Here, the namespaces of 'main', 'i', and 'j', all have references to
'g'. 'i' and 'j' still have them when 'main' loses its.

The lambda might be confusing. Take a simpler example of a namespace:

>>g= [0]
>>def h():

.... class A:
.... a= g
.... return A
....

>>i= h()
>>del g
>>i.a

[0]

Or even:

>>g=[0]
>>class A:

.... a= g
....

>>del g
>>A.a

[0]

The class is executed immediately so 'A.a' gets a hold of 'g'.
Function definitions keep a reference to the -identity-, not value, of
the namespace they were defined in, and parameters are part of that
namespace.

>>def h( a ):

.... #new namespace in here, new on each call
.... def i():
.... return a
.... return i

'i' is not defined in the global namespace, it's defined in a brand
new one that is created on every call of 'h'. 'a' is defined in that,
so that's what 'a' when 'i' is called refers to.

To be specific, the namespace 'h' defines only includes the names that
are different from its enclosing scope. Regardless of what's defined
below 'h', 'h' only defines one new variable, 'a'. Its value is
passed in at call-time. 'i' needs to know what that namespace is--
that's how closures work-- so it saves a reference. That saved
reference is to a namespace, not a variable though, which
distinguishes it from the 'class A: a= g' statement that's executes
immediately. There, 'a' stores the value of g. By contrast, 'i'
stores 'h's entire namespace.

In 'class A: a= g', the name 'a' is assigned to the contents of 'g'.
In 'def i(): return a', 'a' is the value of a look up in a namespace
by its name.

>>def h( a ):

.... #new namespace in here
.... def i():
.... return a
.... return i
....

>>g=[0]
>>j= h(g)
>>hex(id(g))

'0x9ff440'

>>del g
>>hex(id(j()) )

'0x9ff440'

>>j.func_closur e

(<cell at 0x009FDF50: list object at 0x009FF440>,)

By the time 'g' is deleted, 'j' has already hijacked a reference to
it, which lives in the namespace of the 'i' it defined that time
through. 'j()', originally 'g', and 'j's namespace all refer to the
same object.

Variables are keys in a namespace, even if it's an "abandoned
namespace", adopting the term.