nested functions

[Gregory Petrosyan]

I often make helper functions nested, like this:

def f():
def helper():
...
...

is it a good practice or not? What about performance of such
constructs?

--
Regards, Gregory.

[Ben Finney]

Re: nested functions

"Gregory Petrosyan" <gregory.petros yan@gmail.com> writes:
[color=blue]
> I often make helper functions nested, like this:
>
> def f():
> def helper():
> ...
> ...
>
> is it a good practice or not?[/color]

You have my blessing. Used well, it makes for more readable code.
[color=blue]
> What about performance of such constructs?[/color]

What about it? Set up some examples maningful for your situation, with
and without such constructs, and use the profiler to find out.

--
\ "People demand freedom of speech to make up for the freedom of |
`\ thought which they avoid." -- Soren Aabye Kierkegaard |
_o__) (1813-1855) |
Ben Finney

[George Sakkis]

Re: nested functions

Ben Finney wrote:
[color=blue]
> "Gregory Petrosyan" <gregory.petros yan@gmail.com> writes:
>[color=green]
> > I often make helper functions nested, like this:
> >
> > def f():
> > def helper():
> > ...
> > ...
> >
> > is it a good practice or not?[/color]
>
> You have my blessing. Used well, it makes for more readable code.[/color]

I'm not sure it's in general more readable; I typically use nested
functions for closures only, not helper functions, so I'd read the code
twice to check if it's a closure and if not why might have been defined
locally. I prefer to define helpers at the module level, often making
them 'private' by prepending their name with a single underscore.
[color=blue][color=green]
> > What about performance of such constructs?[/color]
>
> What about it? Set up some examples maningful for your situation, with
> and without such constructs, and use the profiler to find out.[/color]

It shouldn't come as a surprise if it turns out to be slower, since the
nested function is redefined every time the outer is called. If you
actually call the outer function a lot, you'd better profile it.

George

[Georg Brandl]

Re: nested functions

George Sakkis wrote:[color=blue]
> Ben Finney wrote:
>[color=green]
>> "Gregory Petrosyan" <gregory.petros yan@gmail.com> writes:
>>[color=darkred]
>> > I often make helper functions nested, like this:
>> >
>> > def f():
>> > def helper():
>> > ...
>> > ...
>> >
>> > is it a good practice or not?[/color]
>>
>> You have my blessing. Used well, it makes for more readable code.[/color]
>
> I'm not sure it's in general more readable; I typically use nested
> functions for closures only, not helper functions, so I'd read the code
> twice to check if it's a closure and if not why might have been defined
> locally. I prefer to define helpers at the module level, often making
> them 'private' by prepending their name with a single underscore.
>[color=green][color=darkred]
>> > What about performance of such constructs?[/color]
>>
>> What about it? Set up some examples maningful for your situation, with
>> and without such constructs, and use the profiler to find out.[/color]
>
> It shouldn't come as a surprise if it turns out to be slower, since the
> nested function is redefined every time the outer is called.[/color]

That's right. However, if the outer function is only called a few times
and the nested function is called a lot, the locals lookup for the
function name is theoretically faster than the globals lookup. Also,
in methods you can use closures, so you don't have to pass, for example,
self to the inner function.

Georg

[Duncan Booth]

Re: nested functions

Georg Brandl wrote:
[color=blue]
> That's right. However, if the outer function is only called a few times
> and the nested function is called a lot, the locals lookup for the
> function name is theoretically faster than the globals lookup. Also,
> in methods you can use closures, so you don't have to pass, for example,
> self to the inner function.[/color]

If you are worried about the overhead of looking up the function name in
the local rather than global scope then you should also worry about the
overhead of accessing self through a closure rather than as a parameter.

As always in these cases, don't worry about it until you know definitely
(by timing) that performance is an issue in that part of your code, and
then time the different options and refuse the temptation to guess as you
will probably get it wrong. The relative times here will depend on a lot of
factors, such as how often you access the closure/parameter, and whether or
not there are other arguments to the function.

I frequently nest functions, but I do it in cases where I want to simplify
a function body and don't see any case for creating yet another generally
accessible method or function.

Some benefits of nested functions: you can use a function name which is
short by self-explanatory within the context of the outer function without
having to worry about it conflicting with other function/variable names.

The extracted functions are kept close to the place where they are used: a
small support function which is a few hundred lines away from where it
is used is more accident prone than one right next to the place it is used.
Also, if you later refactor out the main method the support functions will
disappear as well rather than lying around unused.

You can use closures, not because you have to, but because it simplifies
the calls and therefore keeps expressions simpler and easier to read.

Of course you can also mess things up totally by overdoing it.

[Fredrik Lundh]

Re: nested functions

George Sakkis wrote:
[color=blue]
> It shouldn't come as a surprise if it turns out to be slower, since the
> nested function is redefined every time the outer is called.[/color]

except that it isn't, really: all that happens is that a new function object is created from
prebuilt parts, and assigned to a local variable. it's not slower than, say, a method call.

</F>

[Duncan Booth]

Re: nested functions

Fredrik Lundh wrote:
[color=blue]
> George Sakkis wrote:
>[color=green]
>> It shouldn't come as a surprise if it turns out to be slower, since
>> the nested function is redefined every time the outer is called.[/color]
>
> except that it isn't, really: all that happens is that a new function
> object is created from prebuilt parts, and assigned to a local
> variable. it's not slower than, say, a method call.
>[/color]
It looks to be somewhat faster than a method call:

C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testMethod(t .instance,
42)"
1000 loops, best of 3: 1.58 msec per loop

C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testMethod2( t.instance,
42)"
100 loops, best of 3: 1.61 msec per loop

C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testNested(t .instance,
42)"
1000 loops, best of 3: 1.06 msec per loop

C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testNested2( t.instance,
42)"
1000 loops, best of 3: 1.08 msec per loop

C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testNested3( t.instance,
42)"
1000 loops, best of 3: 1.13 msec per loop

C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testNested4( t.instance,
42)"
1000 loops, best of 3: 1.23 msec per loop


--------- t.py -------------
class C:
def m1(self):
return 42

def m2(self, x):
return x

instance = C()

def testMethod(inst ance,x):
n = 0
while n < 100000:
n += instance.m1()

def testMethod2(ins tance, x):
n = 0
while n < 100000:
n += instance.m2(x)

def testNested(inst ance, x):
def m1():
return 42
n = 0
while n < 100000:
n += m1()

def testNested2(ins tance, x):
def m2():
return x
n = 0
while n < 100000:
n += m2()

def testNested3(ins tance, x):
def m2(y):
return y
n = 0
while n < 100000:
n += m2(x)

def testNested4(ins tance, x):
def m2(y):
return x
n = 0
while n < 100000:
n += m2(x)

----------------------------

The differences between the nested function calls show how difficult it can
be guessing what will be faster: #3&#4 show that all, else being equal,
accessing the closure is much slower than accessing a parameter, but #2
shows that not passing any parameters to the nested function more than
compensates for the single slow closure access.

[George Sakkis]

Re: nested functions

Duncan Booth wrote:
[color=blue]
> Fredrik Lundh wrote:
>[color=green]
> > George Sakkis wrote:
> >[color=darkred]
> >> It shouldn't come as a surprise if it turns out to be slower, since
> >> the nested function is redefined every time the outer is called.[/color]
> >
> > except that it isn't, really: all that happens is that a new function
> > object is created from prebuilt parts, and assigned to a local
> > variable. it's not slower than, say, a method call.
> >[/color]
> It looks to be somewhat faster than a method call:
>
> C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testMethod(t .instance,
> 42)"
> 1000 loops, best of 3: 1.58 msec per loop
>
> C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testMethod2( t.instance,
> 42)"
> 100 loops, best of 3: 1.61 msec per loop
>
> C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testNested(t .instance,
> 42)"
> 1000 loops, best of 3: 1.06 msec per loop
>
> C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testNested2( t.instance,
> 42)"
> 1000 loops, best of 3: 1.08 msec per loop
>
> C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testNested3( t.instance,
> 42)"
> 1000 loops, best of 3: 1.13 msec per loop
>
> C:\temp>\python 24\lib\timeit.p y -s "import t" "t.testNested4( t.instance,
> 42)"
> 1000 loops, best of 3: 1.23 msec per loop
>
>
> --------- t.py -------------
> class C:
> def m1(self):
> return 42
>
> def m2(self, x):
> return x
>
> instance = C()
>
> def testMethod(inst ance,x):
> n = 0
> while n < 100000:
> n += instance.m1()
>
> def testMethod2(ins tance, x):
> n = 0
> while n < 100000:
> n += instance.m2(x)
>
> def testNested(inst ance, x):
> def m1():
> return 42
> n = 0
> while n < 100000:
> n += m1()
>
> def testNested2(ins tance, x):
> def m2():
> return x
> n = 0
> while n < 100000:
> n += m2()
>
> def testNested3(ins tance, x):
> def m2(y):
> return y
> n = 0
> while n < 100000:
> n += m2(x)
>
> def testNested4(ins tance, x):
> def m2(y):
> return x
> n = 0
> while n < 100000:
> n += m2(x)
>
> ----------------------------
>
> The differences between the nested function calls show how difficult it can
> be guessing what will be faster: #3&#4 show that all, else being equal,
> accessing the closure is much slower than accessing a parameter, but #2
> shows that not passing any parameters to the nested function more than
> compensates for the single slow closure access.[/color]

It would also be interesting to add unnested versions of m1(), m2()
(functions, not methods) to the comparison.

George

[Gregory Petrosyan]

Re: nested functions

Thanks everybody for your help!

[Kent Johnson]

Re: nested functions

Fredrik Lundh wrote:[color=blue]
> George Sakkis wrote:
>[color=green]
>> It shouldn't come as a surprise if it turns out to be slower, since the
>> nested function is redefined every time the outer is called.[/color]
>
> except that it isn't, really: all that happens is that a new function object is created from
> prebuilt parts, and assigned to a local variable. it's not slower than, say, a method call.[/color]

Interesting. So func_code for a nested function is created when the
module is compiled, and stuck in a new function object when the
definition is executed. Like George, I always assumed that the body of
the nested function was compiled when the outer function was executed,
but that doesn't really make any sense - the *code* for the inner
function is static, just the environment changes (globals(), closure).

dis.dis reveals all:

In [10]: def g():
....: def h():
....: print 'foo'
....: return h
....:

In [11]: dis.dis(g)
2 0 LOAD_CONST 1 (<code object h at 00E8D960,
file "<ipython console>", line 2>)
3 MAKE_FUNCTION 0
6 STORE_FAST 0 (h)

4 9 LOAD_FAST 0 (h)
12 RETURN_VALUE

Thanks Fredrik!
Kent