Method much slower than function?

**Neil Cerutti** · Jun 14 '07, 01:35 AM

Re: Method much slower than function?

On 2007-06-14, idoerg@gmail.co m <idoerg@gmail.c omwrote:

Hi all,
>
I am running Python 2.5 on Feisty Ubuntu. I came across some code that
is substantially slower when in a method than in a function.
>
############### ## START SOURCE #############
# The function
>
def readgenome(file handle):
s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
s += line.strip()
return s
>
# The method in a class
class bar:
def readgenome(self , filehandle):
self.s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
self.s += line.strip()
>
############### ## END SOURCE ##############
When running the function and the method on a 20,000 line text file, I
get the following:
>

>>>cProfile.run ("bar.readgenom e(open('cb_foo' ))")

20004 function calls in 10.214 CPU seconds
>
Ordered by: standard name
>
ncalls tottime percall cumtime percall
filename:lineno (function)
1 0.000 0.000 10.214 10.214 <string>:1(<mod ule>)
1 10.205 10.205 10.214 10.214 reader.py:11(re adgenome)
1 0.000 0.000 0.000 0.000 {method 'disable' of
'_lsprof.Profil er' objects}
19999 0.009 0.000 0.009 0.000 {method 'strip' of 'str'
objects}
1 0.000 0.000 0.000 0.000 {method 'xreadlines' of
'file' objects}
1 0.000 0.000 0.000 0.000 {open}
>
>

>>>cProfile.run ("z=r.readgenom e(open('cb_foo' ))")

20004 function calls in 0.041 CPU seconds
>
Ordered by: standard name
>
ncalls tottime percall cumtime percall
filename:lineno (function)
1 0.000 0.000 0.041 0.041 <string>:1(<mod ule>)
1 0.035 0.035 0.041 0.041 reader.py:2(rea dgenome)
1 0.000 0.000 0.000 0.000 {method 'disable' of
'_lsprof.Profil er' objects}
19999 0.007 0.000 0.007 0.000 {method 'strip' of 'str'
objects}
1 0.000 0.000 0.000 0.000 {method 'xreadlines' of
'file' objects}
1 0.000 0.000 0.000 0.000 {open}
>
>
The method takes 10 seconds, the function call 0.041 seconds!
>
Yes, I know that I wrote the underlying code rather
inefficiently, and I can streamline it with a single
file.read() call instead if an xreadlines() + strip loop.
Still, the differences in performance are rather staggering!
Any comments?

It is likely the repeated attribute lookup, self.s, that's
slowing it down in comparison to the non-method version.

Try the following simple optimization, using a local variable
instead of an attribute to build up the result.

# The method in a class
class bar:
def readgenome(self , filehandle):
s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
s += line.strip()
self.s = s

To further speed things up, think about using the str.join idiom
instead of str.+=, and using a generator expression instead of an
explicit loop.

# The method in a class
class bar:
def readgenome(self , filehandle):
self.s = ''.join(line.st rip() for line in filehandle)

--
Neil Cerutti

**Gabriel Genellina** · Jun 14 '07, 01:35 AM

Re: Method much slower than function?

En Wed, 13 Jun 2007 21:40:12 -0300, <idoerg@gmail.c omescribió:

Hi all,
>
I am running Python 2.5 on Feisty Ubuntu. I came across some code that
is substantially slower when in a method than in a function.
>
############### ## START SOURCE #############
# The function
>
def readgenome(file handle):
s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
s += line.strip()
return s
>
# The method in a class
class bar:
def readgenome(self , filehandle):
self.s = ''
for line in filehandle.xrea dlines():
if '>' in line:
continue
self.s += line.strip()
>

In the function above, s is a local variable, and accessing local
variables is very efficient (using an array of local variables, the
compiler assigns statically an index for each one).
Using self.s, on the other hand, requires a name lookup for each access.
The most obvious change would be to use a local variable s, and assign
self.s = s only at the end. This should make both methods almost identical
in performance.
In addition, += is rather inefficient for strings; the usual idiom is
using ''.join(items)
And since you have Python 2.5, you can use the file as its own iterator;
combining all this:

return ''.join(line.st rip() for line in filehandle if '>' not in line)

--
Gabriel Genellina

**Grant Edwards** · Jun 14 '07, 04:25 AM

Re: Method much slower than function?

On 2007-06-14, idoerg@gmail.co m <idoerg@gmail.c omwrote:

The method takes 10 seconds, the function call 0.041 seconds!

What happens when you run them in the other order?

The first time you read the file, it has to read it from disk.
The second time, it's probably just reading from the buffer
cache in RAM.

--
Grant Edwards grante Yow! Catsup and Mustard
at all over the place! It's
visi.com the Human Hamburger!

**sjdevnull@yahoo.com** · Jun 14 '07, 04:45 AM

Re: Method much slower than function?

Gabriel Genellina wrote:

In the function above, s is a local variable, and accessing local
variables is very efficient (using an array of local variables, the
compiler assigns statically an index for each one).
Using self.s, on the other hand, requires a name lookup for each access.
The most obvious change would be to use a local variable s, and assign
self.s = s only at the end. This should make both methods almost identical
in performance.

Yeah, that's a big deal and makes a significant difference on my
machine.

In addition, += is rather inefficient for strings; the usual idiom is
using ''.join(items)

Ehh. Python 2.5 (and probably some earlier versions) optimize += on
strings pretty well.

a=""
for i in xrange(100000):
a+="a"

and:

a=[]
for i in xrange(100000):
a.append("a")
a="".join(a)

take virtually the same amount of time on my machine (2.5), and the
non-join version is clearer, IMO. I'd still use join in case I wind
up running under an older Python, but it's probably not a big issue
here.

And since you have Python 2.5, you can use the file as its own iterator;
combining all this:
>
return ''.join(line.st rip() for line in filehandle if '>' not in line)

That's probably pretty good.

**Paul Rubin** · Jun 14 '07, 05:15 AM

Re: Method much slower than function?

"sjdevnull@yaho o.com" <sjdevnull@yaho o.comwrites:

take virtually the same amount of time on my machine (2.5), and the
non-join version is clearer, IMO. I'd still use join in case I wind
up running under an older Python, but it's probably not a big issue here.

You should not rely on using 2.5 or even on that optimization staying in
CPython. Best is to use StringIO or something comparable.

**Gabriel Genellina** · Jun 14 '07, 05:15 AM

Re: Method much slower than function?

En Thu, 14 Jun 2007 01:39:29 -0300, sjdevnull@yahoo .com
<sjdevnull@yaho o.comescribió:

Gabriel Genellina wrote:

>In addition, += is rather inefficient for strings; the usual idiom is
>using ''.join(items)

>
Ehh. Python 2.5 (and probably some earlier versions) optimize += on
strings pretty well.
>
a=""
for i in xrange(100000):
a+="a"
>
and:
>
a=[]
for i in xrange(100000):
a.append("a")
a="".join(a)
>
take virtually the same amount of time on my machine (2.5), and the
non-join version is clearer, IMO. I'd still use join in case I wind
up running under an older Python, but it's probably not a big issue
here.

Yes, for concatenating a lot of a's, sure... Try again using strings
around the size of your expected lines - and make sure they are all
different too.

pyimport timeit
py>
pydef f1():
.... a=""
.... for i in xrange(100000):
.... a+=str(i)*20
....
pydef f2():
.... a=[]
.... for i in xrange(100000):
.... a.append(str(i) *20)
.... a="".join(a)
....
pyprint timeit.Timer("f 2()", "from __main__ import f2").repeat(num ber=1)
[0.4267366383157 6358, 0.4280759146763 0662, 0.4440148119383 8876]
pyprint timeit.Timer("f 1()", "from __main__ import f1").repeat(num ber=1)

....after a few minutes I aborted the process...

--
Gabriel Genellina

**Leo Kislov** · Jun 14 '07, 05:55 AM

Re: Method much slower than function?

On Jun 13, 5:40 pm, ido...@gmail.co m wrote:

Hi all,
>
I am running Python 2.5 on Feisty Ubuntu. I came across some code that
is substantially slower when in a method than in a function.
>

>cProfile.run(" bar.readgenome( open('cb_foo')) ")

>
20004 function calls in 10.214 CPU seconds

>cProfile.run(" z=r.readgenome( open('cb_foo')) ")

>
20004 function calls in 0.041 CPU seconds
>

I suspect open files are cached so the second reader
picks up where the first one left: at the of the file.
The second call doesn't do any text processing at all.

-- Leo

**Bruno Desthuilliers** · Jun 14 '07, 06:55 AM

Re: Method much slower than function?

Neil Cerutti a écrit :
(snip)

class bar:
def readgenome(self , filehandle):
self.s = ''.join(line.st rip() for line in filehandle)

=>
self.s = ''.join(line.st rip() for line in filehandle if not
'>' in line)

**Peter Otten** · Jun 14 '07, 07:35 AM

Re: Method much slower than function?

Gabriel Genellina wrote:

En Thu, 14 Jun 2007 01:39:29 -0300, sjdevnull@yahoo .com
<sjdevnull@yaho o.comescribió:
>

>Gabriel Genellina wrote:

>>In addition, += is rather inefficient for strings; the usual idiom is
>>using ''.join(items)

>>
>Ehh. Python 2.5 (and probably some earlier versions) optimize += on
>strings pretty well.
>>
>a=""
>for i in xrange(100000):
> a+="a"
>>
>and:
>>
>a=[]
>for i in xrange(100000):
> a.append("a")
>a="".join(a)
>>
>take virtually the same amount of time on my machine (2.5), and the
>non-join version is clearer, IMO. I'd still use join in case I wind
>up running under an older Python, but it's probably not a big issue
>here.

>
Yes, for concatenating a lot of a's, sure... Try again using strings
around the size of your expected lines - and make sure they are all
different too.
>
pyimport timeit
py>
pydef f1():
... a=""
... for i in xrange(100000):
... a+=str(i)*20
...
pydef f2():
... a=[]
... for i in xrange(100000):
... a.append(str(i) *20)
... a="".join(a)
...
pyprint timeit.Timer("f 2()", "from __main__ import f2").repeat(num ber=1)
[0.4267366383157 6358, 0.4280759146763 0662, 0.4440148119383 8876]
pyprint timeit.Timer("f 1()", "from __main__ import f1").repeat(num ber=1)
>
...after a few minutes I aborted the process...

I can't confirm this.

$ cat join.py
def f1():
a = ""
for i in xrange(100000):
a += str(i)*20

def f2():
a = []
for i in xrange(100000):
a.append(str(i) *20)
a = "".join(a)

def f3():
a = []
append = a.append
for i in xrange(100000):
append(str(i)*2 0)
a = "".join(a)

$ python2.5 -m timeit -s 'from join import f1' 'f1()'
10 loops, best of 3: 212 msec per loop
$ python2.5 -m timeit -s 'from join import f2' 'f2()'
10 loops, best of 3: 259 msec per loop
$ python2.5 -m timeit -s 'from join import f3' 'f3()'
10 loops, best of 3: 236 msec per loop

Peter

**Peter Otten** · Jun 14 '07, 07:45 AM

Re: Method much slower than function?

Leo Kislov wrote:

On Jun 13, 5:40 pm, ido...@gmail.co m wrote:

>Hi all,
>>
>I am running Python 2.5 on Feisty Ubuntu. I came across some code that
>is substantially slower when in a method than in a function.
>>

>>cProfile.run( "bar.readgenome (open('cb_foo') )")

>>
> 20004 function calls in 10.214 CPU seconds

>

>>cProfile.run( "z=r.readgenome (open('cb_foo') )")

>>
> 20004 function calls in 0.041 CPU seconds
>>

>
I suspect open files are cached so the second reader
picks up where the first one left: at the of the file.
The second call doesn't do any text processing at all.
>
-- Leo

Indeed, the effect of attribute access is much smaller than what the OP is
seeing:

$ cat iadd.py
class A(object):
def add_attr(self):
self.x = 0
for i in xrange(100000):
self.x += 1
def add_local(self) :
x = 0
for i in xrange(100000):
x += 1

add_local = A().add_local
add_attr = A().add_attr
$ python2.5 -m timeit -s 'from iadd import add_local' 'add_local()'
10 loops, best of 3: 21.6 msec per loop
$ python2.5 -m timeit -s 'from iadd import add_attr' 'add_attr()'
10 loops, best of 3: 52.2 msec per loop

Peter

**Francesco Guerrieri** · Jun 14 '07, 09:15 AM

Re: Method much slower than function?

On 6/14/07, Peter Otten <__peter__@web. dewrote:

Gabriel Genellina wrote:

...
pyprint timeit.Timer("f 2()", "from __main__ import f2").repeat(num ber=1)
[0.4267366383157 6358, 0.4280759146763 0662, 0.4440148119383 8876]
pyprint timeit.Timer("f 1()", "from __main__ import f1").repeat(num ber=1)

...after a few minutes I aborted the process...

>
I can't confirm this.

[...]

$ python2.5 -m timeit -s 'from join import f1' 'f1()'
10 loops, best of 3: 212 msec per loop
$ python2.5 -m timeit -s 'from join import f2' 'f2()'
10 loops, best of 3: 259 msec per loop
$ python2.5 -m timeit -s 'from join import f3' 'f3()'
10 loops, best of 3: 236 msec per loop

On my machine (using python 2.5 under win xp) the results are:

>>print timeit.Timer("f 2()", "from __main__ import f2").repeat(num ber = 1)

[0.1972683482282 3575, 0.1932469745640 8974, 0.1947449259421 2861]

>>print timeit.Timer("f 1()", "from __main__ import f1").repeat(num ber = 1)

[21.982707133304 167, 21.905312587963 252, 22.843430035622 767]

so it seems that there is a rather sensible difference.
what's the reason of the apparent inconsistency with Peter's test?

Francesco

**Christof Winter** · Jun 14 '07, 10:45 AM

Re: Method much slower than function?

Gabriel Genellina wrote:
[...]

pyimport timeit
py>
pydef f1():
... a=""
... for i in xrange(100000):
... a+=str(i)*20
...
pydef f2():
... a=[]
... for i in xrange(100000):
... a.append(str(i) *20)
... a="".join(a)
...
pyprint timeit.Timer("f 2()", "from __main__ import f2").repeat(num ber=1)
[0.4267366383157 6358, 0.4280759146763 0662, 0.4440148119383 8876]
pyprint timeit.Timer("f 1()", "from __main__ import f1").repeat(num ber=1)
>
...after a few minutes I aborted the process...
>

Using

Python 2.4.4 (#2, Jan 13 2007, 17:50:26)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2

f1() and f2() also virtually take the same amount of time, although I must admit
that this is quite different from what I expected.

Cheers,
Christof

**Neil Cerutti** · Jun 14 '07, 12:55 PM

Re: Method much slower than function?

On 2007-06-14, Bruno Desthuilliers
<bruno.42.desth uilliers@wtf.we bsiteburo.oops. comwrote:

Neil Cerutti a écrit :
(snip)

>class bar:
> def readgenome(self , filehandle):
> self.s = ''.join(line.st rip() for line in filehandle)

>
>=>
self.s = ''.join(line.st rip() for line in filehandle if not
'>' in line)

Thanks for that correction.

--
Neil Cerutti
I don't know what to expect right now, but we as players have to do what we've
got to do to make sure that the pot is spread equally. --Jim Jackson

**Grant Edwards** · Jun 14 '07, 02:25 PM

Re: Method much slower than function?

On 2007-06-14, Leo Kislov <Leo.Kislov@gma il.comwrote:

On Jun 13, 5:40 pm, ido...@gmail.co m wrote:

>Hi all,
>>
>I am running Python 2.5 on Feisty Ubuntu. I came across some code that
>is substantially slower when in a method than in a function.
>>

>>cProfile.run( "bar.readgenome (open('cb_foo') )")

>>
> 20004 function calls in 10.214 CPU seconds

>

>>cProfile.run( "z=r.readgenome (open('cb_foo') )")

>>
> 20004 function calls in 0.041 CPU seconds
>>

>
I suspect open files are cached

They shouldn't be.

so the second reader picks up where the first one left: at the
of the file.

That sounds like a bug. Opening a file a second time should
produce a "new" file object with the file-pointer at the
beginning of the file.

The second call doesn't do any text processing at all.

--
Grant Edwards grante Yow! I'm using my X-RAY
at VISION to obtain a rare
visi.com glimpse of the INNER
WORKINGS of this POTATO!!

Method much slower than function?

Method much slower than function?

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment