Generated code that is exec-ed (to simulate import) cannot importos.path??

[Irmen de Jong]

Hello,
I don't understand why the following doesn't work.
What I want to do is dynamically import some generated
Python code and I'm doing this using compile and exec'ing
it in the dict of a new empty module object.
That works okay, but as soon as the generated code
tries do perform certain imports, it fails!
Certain other imports succeed. Consider this example code:


-----SNIP----
import imp

source="""
print 'importing random'
import random
print 'importing os'
import os
print 'bye!'

def getName():
return 'Hello there'
"""

newmod = imp.new_module( 'generated.test module')
code=compile(so urce,'<generate d code>','exec')
print 'got code...'
exec code in newmod.__dict__
print 'done, newmod.getname( ): ',newmod.getNam e()
-----/SNIP----


When run, it produces the following output:

[E:\test]python dynimport.py
got code...
importing random
importing os
Traceback (most recent call last):
File "dynimport. py", line 17, in ?
exec code in newmod.__dict__
File "<generated code>", line 5, in ?
File "C:\Python23\li b\os.py", line 131, in ?
from os.path import curdir, pardir, sep, pathsep, defpath, extsep, altsep
ImportError: No module named path


What's going on? Why can't it find os.path?

(Python 2.3.2, tested on linux and windows)

Confused,

Irmen.

[Michael Hudson]

Re: Generated code that is exec-ed (to simulate import) cannot import os.path??

Irmen de Jong <irmen@-NOSPAM-REMOVETHIS-xs4all.nl> writes:
[color=blue]
> Hello,
> I don't understand why the following doesn't work.
> What I want to do is dynamically import some generated
> Python code and I'm doing this using compile and exec'ing
> it in the dict of a new empty module object.
> That works okay, but as soon as the generated code
> tries do perform certain imports, it fails!
> Certain other imports succeed. Consider this example code:[/color]
[snip][color=blue]
> What's going on? Why can't it find os.path?[/color]

Dunno. Two things to try:

1) using new.module instead of the imp function
2) run python -v

Cheers,
mwh

--
CLiki pages can be edited by anybody at any time. Imagine the most
fearsomely comprehensive legal disclaimer you have ever seen, and
double it -- http://ww.telent.net/cliki/index

[Peter Otten]

Re: Generated code that is exec-ed (to simulate import) cannot import os.path??

Irmen de Jong wrote:
[color=blue]
> Hello,
> I don't understand why the following doesn't work.
> What I want to do is dynamically import some generated
> Python code and I'm doing this using compile and exec'ing
> it in the dict of a new empty module object.
> That works okay, but as soon as the generated code
> tries do perform certain imports, it fails!
> Certain other imports succeed. Consider this example code:[/color]

[...]
[color=blue]
> What's going on? Why can't it find os.path?[/color]

My trial and error findings (Python 2.3.2 on Linux):

Python gets confused by the module name "generated.test module", when
a "generated" package does not exist; it seems to look for
"generated.posi xpath" when it should for "os.posixpa th" during the import
of the os module.

Two fixes are possible:

(1) Change

newmod = imp.new_module( 'generated.test module')

to, e. g.

newmod = imp.new_module( 'generated_test module')


(2) Create a dummy "generated" module and insert it into sys.modules:

import imp, sys

source="""
print 'importing random'
import random
print 'importing os'
import os
print 'bye!'

def getName():
return 'Hello there'
"""

sys.modules["generated"] = imp.new_module( "generated" )

newmod = imp.new_module( 'generated.test module')
code=compile(so urce,'<generate d code>','exec')
print 'got code...'
exec code in newmod.__dict__
print 'done, newmod.getname( ): ',newmod.getNam e()

While this works, a helpful error message would be nice when an intermediate
package is not found. Unfortunately I was not able to track down the actual
point of failure yet.

Peter

[Irmen de Jong]

Re: Generated code that is exec-ed (to simulate import) cannot importos.path??

Michael Hudson wrote:[color=blue][color=green]
>>What's going on? Why can't it find os.path?[/color]
>
>
> Dunno. Two things to try:
>
> 1) using new.module instead of the imp function[/color]

Tried it, no difference. I think they are equivalent.
[color=blue]
> 2) run python -v[/color]

A relevant piece of the trace is found below, using
the same code that I posted before. I've done this already
but couldn't find anything that helps me understand why
os.path cannot be imported.


--Irmen


[.... partial 'python -v dynimport.py' output....]
got code...
importing random
# /usr/local/lib/python2.3/random.pyc matches
/usr/local/lib/python2.3/random.py
import generated.rando m # precompiled from
/usr/local/lib/python2.3/random.pyc
dlopen("/usr/local/lib/python2.3/lib-dynload/math.so", 2);
import generated.math # dynamically loaded from
/usr/local/lib/python2.3/lib-dynload/math.so
dlopen("/usr/local/lib/python2.3/lib-dynload/_random.so", 2);
import generated._rand om # dynamically loaded from
/usr/local/lib/python2.3/lib-dynload/_random.so
dlopen("/usr/local/lib/python2.3/lib-dynload/time.so", 2);
import generated.time # dynamically loaded from
/usr/local/lib/python2.3/lib-dynload/time.so
importing os
# /usr/local/lib/python2.3/os.pyc matches /usr/local/lib/python2.3/os.py
import generated.os # precompiled from /usr/local/lib/python2.3/os.pyc
import sys # previously loaded (sys)
import posix # previously loaded (posix)
import posix # previously loaded (posix)
# /usr/local/lib/python2.3/posixpath.pyc matches
/usr/local/lib/python2.3/posixpath.py
import generated.posix path # precompiled from
/usr/local/lib/python2.3/posixpath.pyc
import sys # previously loaded (sys)
# /usr/local/lib/python2.3/stat.pyc matches /usr/local/lib/python2.3/stat.py
import generated.stat # precompiled from /usr/local/lib/python2.3/stat.pyc
import posix # previously loaded (posix)
Traceback (most recent call last):
File "dynimport. py", line 18, in ?
exec code in newmod.__dict__
File "<generated code>", line 5, in ?
File "/usr/local/lib/python2.3/os.py", line 131, in ?
from os.path import curdir, pardir, sep, pathsep, defpath, extsep,
altsep
ImportError: No module named path
# clear __builtin__._
# clear sys.path
# clear sys.argv

[.... end snipped...]

[Irmen de Jong]

Re: Generated code that is exec-ed (to simulate import) cannot importos.path??

Peter Otten wrote:

[color=blue]
> Python gets confused by the module name "generated.test module", when
> a "generated" package does not exist; it seems to look for
> "generated.posi xpath" when it should for "os.posixpa th" during the import
> of the os module.[/color]

Terrific, that was the problem! Thanks a lot Peter!
After changing the module name to 'generated_test module' as you
suggested, it works :-)
Perhaps a more structural improvement is possible but I'm happy
with just changing my generated module name to a name that is
not in a (non-existing) package.
[color=blue]
> While this works, a helpful error message would be nice when an intermediate
> package is not found.[/color]

Indeed...

--Irmen de Jong.