Hey, get this! [was: import from database]

**Peter Otten** · Jul 18 '05, 08:49 PM

Re: Hey, get this! [was: import from database]

Steve Holden wrote:
[color=blue]
> This is even stranger: it makes it if I import the module a second time:[/color]

[second import seems to succeed]

Maybe you are experiencing some version confusion? What you describe looks
much like the normal Python 2.3 behaviour (with no import hook involved)
whereas you seem to operate on the 2.4 library.
A partially initialized module object is left behind in sys.modules and seen
by further import attempts.

$ cat arbitrary.py

import not_there

def f():
print "you ain't gonna see that"

$ python
Python 2.3.3 (#1, Apr 6 2004, 01:47:39)
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> import arbitrary[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary. py", line 2, in ?
import not_there
ImportError: No module named not_there[color=blue][color=green][color=darkred]
>>> import arbitrary
>>> arbitrary.f()[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'module' object has no attribute 'f'[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

I have no experience with import hooks, but for normal imports that has been
fixed in Python 2.4:

$ py24
Python 2.4 (#5, Jan 4 2005, 10:14:01)
[GCC 3.3.3 (SuSE Linux)] on linux2
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> import arbitrary[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary. py", line 2, in ?
import not_there
ImportError: No module named not_there[color=blue][color=green][color=darkred]
>>> import arbitrary[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary. py", line 2, in ?
import not_there
ImportError: No module named not_there[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

Peter

**Steve Holden** · Jul 18 '05, 08:51 PM

Re: Hey, get this! [was: import from database]

Peter Otten wrote:
[color=blue]
> Steve Holden wrote:
>
>[color=green]
>>This is even stranger: it makes it if I import the module a second time:[/color]
>
>
> [second import seems to succeed]
>
> Maybe you are experiencing some version confusion? What you describe looks
> much like the normal Python 2.3 behaviour (with no import hook involved)
> whereas you seem to operate on the 2.4 library.
> A partially initialized module object is left behind in sys.modules and seen
> by further import attempts.
>[/color]
I agree that this is 2.3-like behavior, but Python cannot lie ...

sholden@dellboy ~/Projects/Python/dbimp
$ python
Python 2.4 (#1, Dec 4 2004, 20:10:33)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]
[color=blue]
> $ cat arbitrary.py
>
> import not_there
>
> def f():
> print "you ain't gonna see that"
>
> $ python
> Python 2.3.3 (#1, Apr 6 2004, 01:47:39)
> [GCC 3.3.3 (SuSE Linux)] on linux2
> Type "help", "copyright" , "credits" or "license" for more information.
>[color=green][color=darkred]
>>>>import arbitrary[/color][/color]
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "arbitrary. py", line 2, in ?
> import not_there
> ImportError: No module named not_there
>[color=green][color=darkred]
>>>>import arbitrary
>>>>arbitrary.f ()[/color][/color]
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> AttributeError: 'module' object has no attribute 'f'
>
>
> I have no experience with import hooks, but for normal imports that has been
> fixed in Python 2.4:
>
> $ py24
> Python 2.4 (#5, Jan 4 2005, 10:14:01)
> [GCC 3.3.3 (SuSE Linux)] on linux2
> Type "help", "copyright" , "credits" or "license" for more information.
>[color=green][color=darkred]
>>>>import arbitrary[/color][/color]
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "arbitrary. py", line 2, in ?
> import not_there
> ImportError: No module named not_there
>[color=green][color=darkred]
>>>>import arbitrary[/color][/color]
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "arbitrary. py", line 2, in ?
> import not_there
> ImportError: No module named not_there
>
>
> Peter
>[/color]
$ cat arbitrary.py
import not_there

def f():
print "you ain't gonna see that"

sholden@dellboy ~/Projects/Python/dbimp
$ python
Python 2.4 (#1, Dec 4 2004, 20:10:33)
[GCC 3.3.3 (cygwin special)] on cygwin
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> import arbitrary[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary. py", line 1, in ?
import not_there
ImportError: No module named not_there[color=blue][color=green][color=darkred]
>>> import arbitrary[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "arbitrary. py", line 1, in ?
import not_there
ImportError: No module named not_there[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

Yup, looks like 2.4 (despite this funny cygwin stuff, could that make a
difference).

Let me try it under Windows [ferkle, ferkle ...]

Does the same thing there.

This problem also seems to depend what's already loaded. I wrote a
program to write a test program that looks like this:

import dbimp
dbimp.install()

print "Trying aifc"
try:
import aifc
except:
print "%Failed: aifc"

print "Trying anydbm"
try:
import anydbm
except:
print "%Failed: anydbm"

print "Trying asynchat"
try:
import asynchat
except:
print "%Failed: asynchat"

...

import dbimp
dbimp.install()

print "Trying aifc"
try:
import aifc
except:
print "%Failed: aifc"

print "Trying anydbm"
try:
import anydbm
except:
print "%Failed: anydbm"

print "Trying asynchat"
try:
import asynchat
except:
print "%Failed: asynchat"

The two platforms give expectedly close results. I'm storing compiled
code, so a version incompatibility would be a problem, I agree, but I
have checked the program that loaded the database, and loaded it again
from the Windows source rather than the CygWin source, just to see
whether there were any unnoticed platform dependencies. The results were
exactly the same using either library, and Windows and Cygwin showed
only minor variations.

exhaustCyg24.tx t:%Failed: bsddb.dbtables
exhaustCyg24.tx t:%Failed: bsddb.test.test _all
exhaustCyg24.tx t:%Failed: bsddb.test.test _associate
exhaustCyg24.tx t:%Failed: bsddb.test.test _basics
exhaustCyg24.tx t:%Failed: bsddb.test.test _compat
exhaustCyg24.tx t:%Failed: bsddb.test.test _dbobj
exhaustCyg24.tx t:%Failed: bsddb.test.test _dbshelve
exhaustCyg24.tx t:%Failed: bsddb.test.test _dbtables
exhaustCyg24.tx t:%Failed: bsddb.test.test _env_close
exhaustCyg24.tx t:%Failed: bsddb.test.test _get_none
exhaustCyg24.tx t:%Failed: bsddb.test.test _join
exhaustCyg24.tx t:%Failed: bsddb.test.test _lock
exhaustCyg24.tx t:%Failed: bsddb.test.test _misc
exhaustCyg24.tx t:%Failed: bsddb.test.test _queue
exhaustCyg24.tx t:%Failed: bsddb.test.test _recno
exhaustCyg24.tx t:%Failed: bsddb.test.test _thread
exhaustCyg24.tx t:%Failed: tzparse
exhaustWin24.tx t:%Failed: bsddb.dbtables
exhaustWin24.tx t:%Failed: bsddb.test.test _all
exhaustWin24.tx t:%Failed: bsddb.test.test _associate
exhaustWin24.tx t:%Failed: bsddb.test.test _basics
exhaustWin24.tx t:%Failed: bsddb.test.test _compat
exhaustWin24.tx t:%Failed: bsddb.test.test _dbobj
exhaustWin24.tx t:%Failed: bsddb.test.test _dbshelve
exhaustWin24.tx t:%Failed: bsddb.test.test _dbtables
exhaustWin24.tx t:%Failed: bsddb.test.test _env_close
exhaustWin24.tx t:%Failed: bsddb.test.test _get_none
exhaustWin24.tx t:%Failed: bsddb.test.test _join
exhaustWin24.tx t:%Failed: bsddb.test.test _lock
exhaustWin24.tx t:%Failed: bsddb.test.test _misc
exhaustWin24.tx t:%Failed: bsddb.test.test _queue
exhaustWin24.tx t:%Failed: bsddb.test.test _recno
exhaustWin24.tx t:%Failed: bsddb.test.test _thread
exhaustWin24.tx t:%Failed: pty
exhaustWin24.tx t:%Failed: rlcompleter
exhaustWin24.tx t:%Failed: tzparse

As a workaround I can for the moment just omit everything that show the
least suspicion of not working from the database, but I'd really rather
know what's failing.

If you have any clues I'd be grateful.

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119

**Steve Holden** · Jul 18 '05, 08:59 PM

Re: Hey, get this! [was: import from database]

Steve Holden wrote:
[color=blue]
> Peter Otten wrote:
>[color=green]
>> Steve Holden wrote:
>>
>>[color=darkred]
>>> This is even stranger: it makes it if I import the module a second time:[/color]
>>
>>
>>
>> [second import seems to succeed]
>>
>> Maybe you are experiencing some version confusion? What you describe
>> looks
>> much like the normal Python 2.3 behaviour (with no import hook involved)
>> whereas you seem to operate on the 2.4 library.
>> A partially initialized module object is left behind in sys.modules
>> and seen
>> by further import attempts.
>>[/color]
> I agree that this is 2.3-like behavior, but Python cannot lie ...
>
> sholden@dellboy ~/Projects/Python/dbimp
> $ python
> Python 2.4 (#1, Dec 4 2004, 20:10:33)
> [GCC 3.3.3 (cygwin special)] on cygwin
> Type "help", "copyright" , "credits" or "license" for more information.[color=green][color=darkred]
> >>>[/color][/color]
>[/color]
Just to make things simpler, and (;-) to appeal to a wider audience,
here is a program that doesn't use database at all (it loads the entire
standard library into a dict) and still shows the error.

What *I* would like to know is: who is allowing the import of bsddb.os,
thereby somehow causing the code of the os library module to be run a
second time.

#
# Establish standard library in dict
#
import os
import glob
import sys
import marshal
import new

def importpy(dct, path, modname, package):
c = compile(file(pa th).read(), path, "exec")
dct[modname] = (marshal.dumps( c), package, path)
if package:
print "Package", modname, path
else:
print "Module", modname, path

def importall(dct, path, modlist):
os.chdir(path)
for f in glob.glob("*"):
if os.path.isdir(f ):
fn = os.path.join(pa th, f, "__init__.p y")
if os.path.exists( fn):
ml = modlist + [f]
importpy(dct, fn, ".".join(ml ), 1)
importall(dct, os.path.join(pa th, f), ml)
elif f.endswith('.py ') and '.' not in f[:-3] and f !=
"__init__.p y":
importpy(dct, os.path.join(pa th, f),
".".join(modlis t+[f[:-3]]), 0)

class dbimporter(obje ct):

def __init__(self, item, *args, **kw):
##print "dbimporter : item:", item, "args:", args, "keywords:" , kw
if item != "*db*":
raise ImportError
print "Accepted", item

def find_module(sel f, fullname, path=None):
print "find_modul e:", fullname, "from", path
if fullname not in impdict:
#print "Bailed on", fullname
return None
else:
print "found", fullname, "in db"
return self

def load_module(sel f, modname):
print "load_modul e:", modname
if modname in sys.modules:
return sys.modules[modname]
try:
row = impdict[modname]
except KeyError:
#print modname, "not found in db"
raise ImportError, "DB module %s not found in modules"
code, package, path = row
code = marshal.loads(c ode)
module = new.module(modn ame)
sys.modules[modname] = module
module.__name__ = modname
module.__file__ = path # "db:%s" % modname
module.__loader __ = dbimporter
if package:
module.__path__ = sys.path
exec code in module.__dict__
print modname, "loaded:", repr(module), "pkg:", package
return module

def install():
sys.path_hooks. append(dbimport er)
sys.path_import er_cache.clear( ) # probably not necessary
sys.path.insert (0, "*db*") # probably not needed with a metea-path
hook?

if __name__ == "__main__":
impdict = {}
for path in sys.argv[1:]:
importall(impdi ct, path, [])
install()
import bsddb

Running this against a copy of the Python 2.4 standard library in C:\Lib
gives me

[...]
Module _strptime C:\Lib\_strptim e.py
Module _threading_loca l C:\Lib\_threadi ng_local.py
Module __future__ C:\Lib\__future __.py
Accepted *db*
find_module: bsddb from None
found bsddb in db
load_module: bsddb
find_module: bsddb._bsddb from None
find_module: bsddb.sys from None
find_module: bsddb.os from None
find_module: bsddb.nt from None
find_module: bsddb.ntpath from None
find_module: bsddb.stat from None
Traceback (most recent call last):
File "C:\Steve\Proje cts\Python\dbim p\dictload.py", line 79, in ?
import bsddb
File "C:\Steve\Proje cts\Python\dbim p\dictload.py", line 65, in
load_module
exec code in module.__dict__
File "C:\Lib\bsddb\_ _init__.py", line 62, in ?
import sys, os
File "C:\Python24\li b\os.py", line 133, in ?
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
altsep,
ImportError: No module named path

The 2.3 bsddb library doesn't cause the same problems (and even loads
into 2.4 quite nicely). Lots of modules *will* import, and most packages
don't seem to cause problems. Anyone give me a pointer here?

regards
Steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.python.org/pycon/2005/
Steve Holden http://www.holdenweb.com/

**Bernhard Herzog** · Jul 18 '05, 08:59 PM

Re: Hey, get this!

Steve Holden <steve@holdenwe b.com> writes:
[color=blue]
> What *I* would like to know is: who is allowing the import of bsddb.os,
> thereby somehow causing the code of the os library module to be run a
> second time.[/color]

I would guess (without actually running the code) that this part is
responsible:
[color=blue]
> if package:
> module.__path__ = sys.path[/color]

You usually should initialize a package's __path__ to an empty list.
The __init__ module will take care of modifying it if necessary.

Bernhard

--
Intevation GmbH http://intevation.de/
Skencil http://skencil.org/
Thuban http://thuban.intevation.org/

**Steve Holden** · Jul 18 '05, 08:59 PM

Re: Hey, get this!

Bernhard Herzog wrote:
[color=blue]
> Steve Holden <steve@holdenwe b.com> writes:
>
>[color=green]
>>What *I* would like to know is: who is allowing the import of bsddb.os,
>>thereby somehow causing the code of the os library module to be run a
>>second time.[/color]
>
>
> I would guess (without actually running the code) that this part is
> responsible:
>
>[color=green]
>> if package:
>> module.__path__ = sys.path[/color]
>
>
> You usually should initialize a package's __path__ to an empty list.
> The __init__ module will take care of modifying it if necessary.
>
> Bernhard
>[/color]
Thanks, Bernhard, that appeared to fix the problem. The error certainly
went away, anyway ... now I can do more experimentation .

regards
Steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.python.org/pycon/2005/
Steve Holden http://www.holdenweb.com/

Hey, get this! [was: import from database]

Hey, get this! [was: import from database]

Comment

Comment

Comment

Comment

Comment