module import performance question

[Jeff Sykes]

I have a cgi script that imports modules based on a user config file.
I am suffering some performance problems when I import these modules.
Some logging revealed that it seems to be taking about 1.3 seconds to
import these modules. This is running on a Windows 2000 box with a
Pentium II 400 processor, Python 2.1, Apache 1.3.19 as the web server.
I don't need screaming performance numbers, but this time is
excessive. Is it me, or does that number seem kind of slow? Any code
optimizations or recommendations ? I would like to keep the
architecture the same, just standard cgi scripts, so mod_python looks
like it's a bit more involved than I would like.

Here is a code snip:

dyn_blocks = {} # set up a dict of the dynamic imports
if confok: # was the configuration present?
blocks = getBlocks(confi g) # get the application info from config
for block in blocks: # each block indictates the ap & pg (module)
ap = string.split(st ring.strip(bloc k[1:len(block)]))[0]
pg = string.split(st ring.strip(bloc k[1:len(block)]))[1]
sys.path.append (getAppPath(ap) ) # add module dir to sys.path
if (block[0] == '~'):
try:
dyn_blocks[ap + "_" + pg] = __import__(pg)
except:
Log.stacktrace( Log.ERROR) # log problem importing module

Any help is appreciated.

[Emile van Sebille]

Re: module import performance question


"Jeff Sykes" <jeff@cowz.co m> wrote in message
news:729b56c7.0 311102210.4869d 4f6@posting.goo gle.com...[color=blue]
> I have a cgi script that imports modules based on a user config[/color]
file.[color=blue]
> I am suffering some performance problems when I import these[/color]
modules.[color=blue]
> Some logging revealed that it seems to be taking about 1.3 seconds[/color]
to[color=blue]
> import these modules. This is running on a Windows 2000 box with a
> Pentium II 400 processor, Python 2.1, Apache 1.3.19 as the web[/color]
server.[color=blue]
> I don't need screaming performance numbers, but this time is
> excessive. Is it me, or does that number seem kind of slow?[/color]

How many modules are being imported?
[color=blue]
> Any code
> optimizations or recommendations ?[/color]

Don't append to sys.path. Try sys.path.insert (0,addlpath). Then the
import will get an immediate hit and always the right module rather
than stepping through all the preceeding directories.

HTH,


Emile van Sebille
emile@fenx.com

[Jeff Sykes]

Re: module import performance question

Thanks, Emile, that's a good idea. I found another issue, too. See my
response to another poster below:
[color=blue][color=green]
>> I think there was a different culprit, though. After some more
>> dilligent logging, I realized that immediately after import I
>> called upon an attribute of the imported module that wouldn't
>> _necessarily_ be present. When the attribute was not present,
>> it raised an exception, which I caught. This exception handling
>> was a real performance anchor! I now check hasattr() before I
>> execute this block of code, and this has helped performance
>> significantly.
>>
>> Here's a snip of the block that occured right after import:
>>
>> try:
>> # next line was not there before
>> if hasattr(dyn_blo cks[key], "BUFFER_ON" ):
>> context.setBuff erFlag(dyn_bloc ks[key].BUFFER_ON)
>> except:
>> Log.stacktrace( Log.ERROR)
>>
>> I guess the lesson I learned is don't use exception handling
>> unnecessarily. Which I know not to do anyway. Sigh. Back to
>> school for me.[/color][/color]

"Emile van Sebille" <emile@fenx.com > wrote in message news:<boqutc$1g l3q2$1@ID-11957.news.uni-berlin.de>...[color=blue]
> "Jeff Sykes" <jeff@cowz.co m> wrote in message
> news:729b56c7.0 311102210.4869d 4f6@posting.goo gle.com...[color=green]
> > I have a cgi script that imports modules based on a user config[/color]
> file.[color=green]
> > I am suffering some performance problems when I import these[/color]
> modules.[color=green]
> > Some logging revealed that it seems to be taking about 1.3 seconds[/color]
> to[color=green]
> > import these modules. This is running on a Windows 2000 box with a
> > Pentium II 400 processor, Python 2.1, Apache 1.3.19 as the web[/color]
> server.[color=green]
> > I don't need screaming performance numbers, but this time is
> > excessive. Is it me, or does that number seem kind of slow?[/color]
>
> How many modules are being imported?
>[color=green]
> > Any code
> > optimizations or recommendations ?[/color]
>
> Don't append to sys.path. Try sys.path.insert (0,addlpath). Then the
> import will get an immediate hit and always the right module rather
> than stepping through all the preceeding directories.
>
> HTH,
>
>
> Emile van Sebille
> emile@fenx.com[/color]

[Peter Abel]

Re: module import performance question

jeff@cowz.com (Jeff Sykes) wrote in message news:<729b56c7. 0311102210.4869 d4f6@posting.go ogle.com>...
....
....
....
[color=blue]
> Here is a code snip:
>
> dyn_blocks = {} # set up a dict of the dynamic imports
> if confok: # was the configuration present?
> blocks = getBlocks(confi g) # get the application info from config
> for block in blocks: # each block indictates the ap & pg (module)[/color]

doen't know how long is len(blocks), if it's very long you could
accelerate the *for loop* be replacing the following two lines with:

(ap,pg) = string.split(st ring.strip(bloc k[1:]))
**Remark: block[1:len(block)] is the same as block[1:], but without
calulating len(block) everytime, further more the split
and the strip is done only once
[color=blue]
> ap = string.split(st ring.strip(bloc k[1:len(block)]))[0]
> pg = string.split(st ring.strip(bloc k[1:len(block)]))[1][/color]

....
....
....

Regards
Peter

[Scott D. Daniels]

Re: module import performance question

jeff@cowz.com (Jeff Sykes) wrote in message news:<729b56c7. 0311111158.3602 e7d5@posting.go ogle.com>...[color=blue]
> ...
> Here's a snip of the block that occured right after import:
>
> try:
> # next line was not there before
> if hasattr(dyn_blo cks[key], "BUFFER_ON" ):
> context.setBuff erFlag(dyn_bloc ks[key].BUFFER_ON)
> except:
> Log.stacktrace( Log.ERROR)
> I guess the lesson I learned is don't use exception handling
> unnecessarily. Which I know not to do anyway. Sigh. Back to
> school for me.[/color]

Wrong lesson. Python wants you to use exception handling.
The time sink above code is the unnecessary uses of:[color=blue]
> Log.stacktrace( Log.ERROR)[/color]

A couple of Alternatives:
## smaller, not quite careful enough for my taste
try:
context.setBuff erFlag(dyn_bloc ks[key].BUFFER_ON)
except AttributeError:
pass
except:
Log.stacktrace( Log.ERROR)

## better, more to my taste
try:
try:
flag = dyn_blocks[key].BUFFER_ON
except AttributeError:
pass
else:
context.setBuff erFlag(flag)
except:
Log.stacktrace( Log.ERROR)

-Scott