app runs fine with interpreter, but not under py2exe

[Doug Morse]

Hi,

I have an application that runs just fine using the standard Python distro
interpreter (v2.5.1 on WinXP) but throws the following exception when run as
an executable built with py2exe. My questions are: (a) does anyone have any
thoughts on why this exception is occurring and what to do about it, and (b)
more generally, why would a problem like this occur under py2exe but not with
the standard distro?

Thanks in adavance for any and all help.

Cheers,
Doug


Traceback (most recent call last):
File "VisionTrainer. py", line 49, in <module>
File "SessionControl ler.pyc", line 53, in <module>
File "VisionEgg\__in it__.pyc", line 42, in <module>
File "VisionEgg\Para meterTypes.pyc" , line 28, in <module>
File "Numeric.py c", line 93, in <module>
File "Precision.pyc" , line 26, in <module>
File "Precision.pyc" , line 23, in _fill_table
File "Precision.pyc" , line 18, in _get_precisions
TypeError: data type not understood

[GHUM]

Re: app runs fine with interpreter, but not under py2exe

Doug,

File "VisionTrainer. py", line 49, in <module>
File "SessionControl ler.pyc", line 53, in <module>
File "VisionEgg\__in it__.pyc", line 42, in <module>
File "VisionEgg\Para meterTypes.pyc" , line 28, in <module>
File "Numeric.py c", line 93, in <module>
File "Precision.pyc" , line 26, in <module>
File "Precision.pyc" , line 23, in _fill_table
File "Precision.pyc" , line 18, in _get_precisions
TypeError: data type not understood

to my knowledge, "data type not understood" is a message not from core
Python, but rather thrown from "your" code. What is happening around
Precision.py, line 18?

What does trigger that exception?

My guess is, you are dealing with some custom imported modules which
define "data type"s.
(PIL does something similiar for supported images)... that is, some
module is trying to import all definitions from a specific directory.

That "dynamic importing" fails within py2exe --- all the importing has
to be definit at py2exing-time. Please ready http://www.py2exe.org/index.cgi/PIL_and_py2exe
and the other receipe on py2exe.org and try to adapt that knowledge to
your application.

Harald

Harald

[GHUM]

Re: app runs fine with interpreter, but not under py2exe

Doug,

Precision.py is part of the Numeric package. AFAIKT, the problem is during
the module initialization. The first lines of Precision.py are:
>
from multiarray import zeros
import string
>

[.....]
and your program is crashing on the

lst.append( (zeros( (1,), t ).itemsize()*8, t) ) <-- Line 18

line... Please, try the following:

import multiarray
from multiarray import zeros
import string

(adding the line "import multiarray" before zeros are imported from
it)

I used this workaround with various packages I py2exed - maybe it also
works for you?

please keep us updated,

Harald

[Peter Otten]

Re: app runs fine with interpreter, but not under py2exe

Doug Morse wrote:

from multiarray import zeros
import string
>
typecodes = {'Character':'c ', 'Integer':'1sil ', 'UnsignedIntege r':'bwu',
'Float':'fd', 'Complex':'FD'}
>
def _get_precisions (typecodes):
    lst = []
    for t in typecodes:
        lst.append( (zeros( (1,), t ).itemsize()*8, t) )   <-- Line 18
    return lst
>
def _fill_table(typ ecodes, table={}):
    for key, value in typecodes.items ():
        table[key] = _get_precisions (value)
    return table
>
_code_table = _fill_table(typ ecodes)
>
I can't find any reason why line 18 is throwing a "data type not
understood" error.  It doesn't seem to have anything to do with dynamic
importing, but I can't be sure.  I would note that "zeros" is a built-in
function found in the "python dll" multiarray.pyd (in the Numeric module
directory).

I don't know why, but your module seems to use numpy's multiarray instead of
Numeric's:

>>from multiarray import zeros
>>zeros((1,), "1")

array([0], '1')

>>from numpy.core.mult iarray import zeros
>>zeros((1,), "1")

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: data type not understood

Peter

[Doug Morse]

Re: app runs fine with interpreter, but not under py2exe

Hi,

Well, my attempt to not use the --skip-archive option didn't get very far,
as I quickly noticed that "library.zi p" does NOT contain ANY .pyd files.
I'm guessing that they can't be in library.zip for a reason (i.e., they are
DLL files, essentially, and thus must be readily available to be loaded
into memory).

Is there a work-around for this, then? That is, is there a way to either
(a) tell py2exe how to *correctly* handle multiple multiarray.pyd and
umath.pyd files or (b) perhaps rename one set of the .pyd files -- say the
numpy/core versions -- to say multiarray2.pyd and umath2.pyd, and then
manual create the "stub"-like .pyc files that py2exe creates to point to
these alternate .pyd files and then place these stubs in
library.zip/numpy/core? Or am I just hoping for too much here and am going
to be stuck with using the --skip-archive option?

Thanks,
Doug


On Fri, 14 Mar 2008 14:37:32 +0000 (UTC), Doug Morse <morse@edoug.or gwrote:

Peter,
>
Genius! You nailed it -- thanks!
>
py2exe is apparently getting confused by the fact that packages "Numeric"
...
<snip>
>
So, my next step will be to try to not use the --skip-archive option and
then make these same modifications regarding multiarray.pyd and umath.pyd
to the py2exe-generated library.zip file and see if I can get things
running that way as well (and in so doing reduce my "dist" directory by
about 10mg). I may also try creating a dist/Numeric subdirectory and
moving dist/multiarray.pyd and dist/umath.pyd to this dist/Numeric
subdirectory -- for the goal of more accurately mirroring the actual
file/directory structure found in $PYTHONHOME/Lib/site-packages.
>
<snip>

[Thomas Heller]

Re: app runs fine with interpreter, but not under py2exe

Doug Morse schrieb:

Peter,
>
Genius! You nailed it -- thanks!
>
py2exe is apparently getting confused by the fact that packages "Numeric" and
"numpy" both have files multiarray.pyd and umath.pyd. It copies just one of
each -- from $PYTHONHOME/Lib/site-packages/numpy/core -- and puts both of them
into the top-level of the created "dist" directory. Also, there are no such
files as multiarray.pyc and umath.pyc in my python installation, but py2exe
seems to create these (in the dist and dist/numpy/core directories) -- they
are small (e.g., 526 bytes) and I'm guessing that they are stubs that simply
point python to the matching .pyd that py2exe relocated.

[...]

>
Just for completeness and perhaps a bit of additional clarity, here's a
limited directory listing of what the steps in the previous paragraph produce:
>
morsefind dist | egrep "(multia|umath) " | xargs ls -l
-rwxr-xr-x 1 morse None 26624 Nov 28 2006 dist/multiarray.pyd
-rwxr-xr-x 1 morse None 348160 Nov 8 16:16 dist/numpy/core/multiarray.pyd
-rwxr-xr-x 1 morse None 192512 Nov 8 16:16 dist/numpy/core/umath.pyd
-rwxr-xr-x 1 morse None 54272 Nov 28 2006 dist/umath.pyd
>

[...]

Is this something I should pursue getting the py2exe folks to fix / work with
them on fixing? In investigating this issue, I noted that a lot of other
py2exe problems seem to revolve around confusions when duplicate files exist
in different modules. I wouldn't thinking that getting py2exe to pay better
attention to the containing modules when building it's dependency tree would
be that difficult (or is it)?

I have the impression that this issue may already be fixed in py2exe CVS but
noone bothered to do a release. Here is the corresponding changelog entry:

* build_exe.py: Patch from Grant Edwards, slightly adjusted:
py2exe now renames the pyd files that it copies into the dist
directory so that they include the package name. This prevents
name conflicts.

I have created installers and uploaded them (for testing!) to the starship:

http://starship.python.net/crew/theller/py2exe-0.6.7.win32-py2.3.exe

http://starship.python.net/crew/theller/py2exe-0.6.7.win32-py2.4.exe

http://starship.python.net/crew/theller/py2exe-0.6.7.win32-py2.5.exe

Please try them out.

Thanks,
Thomas