Problem building/using M2Crypto extension module

[Carl Waldbieser]

I tried to adapt the instructions for building the M2Crypto module
(http://sandbox.rulemaker.net/ngps/m2/INSTALL.html) to build a version
compatible with Python2.3, but I've had some mixed results. I actually got
everything to build and install, but when I try to import M2Crypto-- well,
here is a sample session:
[color=blue][color=green][color=darkred]
>>> import M2Crypto[/color][/color][/color]
Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
File "C:\PYTHON23\Li b\site-packages\M2Cryp to\__init__.py" , line 7, in ?
import __m2crypto
ImportError: DLL load failed: The system cannot find the file specified.[color=blue][color=green][color=darkred]
>>> import M2Crypto
>>> dir()[/color][/color][/color]
['M2Crypto', '__builtins__', '__doc__', '__name__', 'pywin'][color=blue][color=green][color=darkred]
>>>[/color][/color][/color]
[color=blue][color=green][color=darkred]
>>> import M2Crypto.Rand[/color][/color][/color]
Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
File "C:\PYTHON23\Li b\site-packages\M2Cryp to\Rand.py", line 7, in ?
import m2
File "C:\PYTHON23\Li b\site-packages\M2Cryp to\m2.py", line 7, in ?
from __m2crypto import *
ImportError: DLL load failed: The system cannot find the file specified.[color=blue][color=green][color=darkred]
>>> import M2Crypto.Rand
>>>[/color][/color][/color]

As you can see, the first time I try to import the module, it fails, but the
next time, it works. The same thing happens when I try to import a
sub-module underneath it.

Has anyone experienced any problems similar to this when building extension
modules? Is there some sort of common trap I might have fallen into?

The following is version info:
I am running Python 2.3.1 on Windows98. I built the extension from source,
using the mingw compiler 3.2.3. I built version 0.9-7c of the OpenSSL
library to use with M2Crypto. This is the first time I used SWIG 1.3.19 and
PExport 0.42h.

This is the first time I have actually tried to build a Python extension on
this PC, so the chances are probably pretty good that I have screwed
something up, but I am not sure how I would go about tracking it down.

If anyone else has successfully built and used this extension for Python2.3
on Windows, I would be glad to hear it, just to know it is possible. If
anyone has any advice or suggestions, I would also be glad to hear that,
too!

Thanks,
Carl Waldbieser

[Ng Pheng Siong]

Re: Problem building/using M2Crypto extension module

According to Carl Waldbieser <waldbie@attglo bal.net>:[color=blue][color=green][color=darkred]
> >>> import M2Crypto[/color][/color]
> Traceback (most recent call last):
> File "<interacti ve input>", line 1, in ?
> File "C:\PYTHON23\Li b\site-packages\M2Cryp to\__init__.py" , line 7, in ?
> import __m2crypto
> ImportError: DLL load failed: The system cannot find the file specified.[/color]

Is there a __m2crypto.dll or __m2crypto.pyd (sorry can't remember which)
somewhere?

I haven't tried with Python 2.3 on Windows, but I've done it on FreeBSD. No
change in build procedure.

$ python
Python 2.3 (#3, Sep 15 2003, 15:22:12)
[GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> import M2Crypto
>>>[/color][/color][/color]

$ python2.3 alltests.py
............... ............... ............... ............... ..........
----------------------------------------------------------------------
Ran 70 tests in 6.756s

OK


On Windows, I use Bloodshed DevC++, which is some version of mingw. Instead
of pydef, I use the free BC++'s impdef, but either should produce
equivalent .def files.

I was going to install Python 2.3 for Windows, then I noticed 2.3.1 and
then I noticed 2.3.2 release candidate, so I decided to wait a little.

Do you have Python 2.2.3, say? Are you able to build on this version?


--
Ng Pheng Siong <ngps@netmemeti c.com>

http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
http://sandbox.rulemaker.net/ngps -+- Open Source Python Crypto & SSL

[Carl Waldbieser]

Re: Problem building/using M2Crypto extension module

There is a __m2crypto.pyd in my \Python23\Lib\s ite-packages\M2Cryp to
directory. I don't have Python 2.2.3 installed, but I may eventually
re-install it just to see if I can get this working.

Is there some way I can inspect the output files at each step of the build
so as to detect an error in my process? Also, I do not recall using pydef,
and I am not sure what it is. I beilieve that I used a tool called pexport
to make the .def file for python.

If you get around to building it for Python2.3.x, I'd be glad to know how it
went. One thing I got hung up on for a long time was trying to get the
distutils to do the install. If I tried:
[color=blue]
>python setup.py install[/color]

I would get an error because I built the files using mingw and it would say
I needed to build it with Visual C++. Of course, I already had built it for
mingw with:
[color=blue]
>python setup.py build -cmingw[/color]

so I was a bit confused. There was no such flag for the install, as far as
I could tell. After reading through the help for a while, I found a flag
that let me skip the re-build that is implicit in the install. I didn't
think that should have caused a problem, though.

Carl Waldbieser

"Ng Pheng Siong" <ngps@netmemeti c.com> wrote in message
news:blhhc4$sgp $3@mawar.singne t.com.sg...[color=blue]
> According to Carl Waldbieser <waldbie@attglo bal.net>:[color=green][color=darkred]
> > >>> import M2Crypto[/color]
> > Traceback (most recent call last):
> > File "<interacti ve input>", line 1, in ?
> > File "C:\PYTHON23\Li b\site-packages\M2Cryp to\__init__.py" , line 7, in[/color][/color]
?[color=blue][color=green]
> > import __m2crypto
> > ImportError: DLL load failed: The system cannot find the file specified.[/color]
>
> Is there a __m2crypto.dll or __m2crypto.pyd (sorry can't remember which)
> somewhere?
>
> I haven't tried with Python 2.3 on Windows, but I've done it on FreeBSD.[/color]
No[color=blue]
> change in build procedure.
>
> $ python
> Python 2.3 (#3, Sep 15 2003, 15:22:12)
> [GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
> Type "help", "copyright" , "credits" or "license" for more information.[color=green][color=darkred]
> >>> import M2Crypto
> >>>[/color][/color]
>
> $ python2.3 alltests.py
> ............... ............... ............... ............... ..........
> ----------------------------------------------------------------------
> Ran 70 tests in 6.756s
>
> OK
>
>
> On Windows, I use Bloodshed DevC++, which is some version of mingw.[/color]
Instead[color=blue]
> of pydef, I use the free BC++'s impdef, but either should produce
> equivalent .def files.
>
> I was going to install Python 2.3 for Windows, then I noticed 2.3.1 and
> then I noticed 2.3.2 release candidate, so I decided to wait a little.
>
> Do you have Python 2.2.3, say? Are you able to build on this version?
>
>
> --
> Ng Pheng Siong <ngps@netmemeti c.com>
>
> http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
> http://sandbox.rulemaker.net/ngps -+- Open Source Python Crypto & SSL[/color]

[Ng Pheng Siong]

Re: Problem building/using M2Crypto extension module

According to Carl Waldbieser <waldbie@attglo bal.net>:[color=blue]
> There is a __m2crypto.pyd in my \Python23\Lib\s ite-packages\M2Cryp to
> directory.[/color]

cd there, import the DLL into the interpreter toplevel:

$ ls -l __m2crypto.so
-rwxr-xr-x 1 ngps ngps 458146 Sep 28 20:59 __m2crypto.so*

$ python
Python 2.3 (#3, Sep 15 2003, 15:22:12)
[GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> import __m2crypto
>>>[/color][/color][/color]

Does it work?
[color=blue]
> Is there some way I can inspect the output files at each step of the build
> so as to detect an error in my process? Also, I do not recall using pydef,
> and I am not sure what it is. I beilieve that I used a tool called pexport
> to make the .def file for python.[/color]

Sorry, typo, I meant pyexport.

I've just put up 0.12 snapshot #1. There is a SWIG/Makefile.mw, which is
for use with mingw. Take a look and adjust paths (e.g., OpenSSL
include/library files, swig.exe) where necessary. Try it with "make -f
Makefile.mw".


--
Ng Pheng Siong <ngps@netmemeti c.com>

http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
http://sandbox.rulemaker.net/ngps -+- Open Source Python Crypto & SSL

[Ng Pheng Siong]

Re: Problem building/using M2Crypto extension module

According to Carl Waldbieser <waldbie@attglo bal.net>:[color=blue]
> If you get around to building it for Python2.3.x, I'd be glad to know how it
> went.[/color]

Turns out I do have Python 2.3 installed on one of my
rarely-booted-into-Windows computer.

Here's what I get:

C:\ngps\prog\ng ps\m2\SWIG>make -f Makefile.mw
c:/pkg/swig/swig.exe -shadow -python _m2crypto.i
gcc -c -DTHREADING -g -Ic:/pkg/py23/include -Ic:/pkg/openssl/include -I. _m2cryp
to_wrap.c
[warnings deleted]
dllwrap --dllname __m2crypto.pyd --driver-name gcc \
--def _m2crypto.def -o __m2crypto.pyd _m2crypto_wrap. o \
-s --entry _DllMain@12 --target=i386-mingw32 c:/pkg/py23/libs/libpyt
hon23.a c:/pkg/openssl/lib/libssl32.a c:/pkg/openssl/lib/libeay32.a
copy _m2crypto.py ..\M2Crypto
1 file(s) copied.
copy __m2crypto.pyd ..\M2Crypto
1 file(s) copied.

C:\ngps\prog\ng ps\m2\SWIG>pyth on
Python 2.3 (#46, Jul 29 2003, 18:54:32) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> import __m2crypto
>>>[/color][/color][/color]


C:\ngps\prog\ng ps\m2>python setup.py build -cmingw32
running build
running build_py
running build_ext
building '__m2crypto' extension
swigging SWIG/_m2crypto.i to SWIG/_m2crypto_wrap. c
c:\pkg\swig\swi g.exe -python -ISWIG -o SWIG/_m2crypto_wrap. c SWIG/_m2crypto.i
g:\pkg\dev-c++\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\ngps\prog\n gps\m2\SWI
G -Ic:\pkg\openssl/include -Ic:\pkg\py23\in clude -Ic:\pkg\py23\PC -c SWIG/_m2cry
pto_wrap.c -o build\temp.win3 2-2.3\Release\swi g\_m2crypto_wra p.o -DTHREADING
[warnings deleted]
writing build\temp.win3 2-2.3\Release\swi g\__m2crypto.de f
g:\pkg\dev-c++\bin\dllwrap .exe -mno-cygwin -mdll -static --output-lib build\temp
..win32-2.3\Release\swi g\lib__m2crypto .a --def build\temp.win3 2-2.3\Release\swi g\
__m2crypto.def -s build\temp.win3 2-2.3\Release\swi g\_m2crypto_wra p.o -Lc:\pkg\op
enssl\lib -Lc:\pkg\py23\li bs -Lc:\pkg\py23\PC Build -lssleay32 -llibeay32 -lpytho
n23 -o build\lib.win32-2.3\M2Crypto\__ m2crypto.pyd

C:\ngps\prog\ng ps\m2>python setup.py install --skip-build
running install
running install_lib
creating c:\pkg\py23\Lib \site-packages\M2Cryp to
copying build\lib.win32-2.3\M2Crypto\AS N1.py -> c:\pkg\py23\Lib \site-packages\M2
Crypto
[blah blah blah]
byte-compiling c:\pkg\py23\Lib \site-packages\M2Cryp to\__init__.py to __init__.py
c

C:\>python
Python 2.3 (#46, Jul 29 2003, 18:54:32) [MSC v.1200 32 bit (Intel)] on win32
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> import M2Crypto
>>> dir(M2Crypto)[/color][/color][/color]
['ASN1', 'BIO', 'DH', 'DSA', 'EVP', 'Err', 'PGP', 'RC4', 'RCS_id', 'RSA', 'Rand'
, 'SSL', 'X509', '__builtins__', '__doc__', '__file__', '__m2crypto', '__name__'
, '__path__', 'decrypt', 'encrypt', 'httpslib', 'm2', 'm2urllib', 'urllib2', 'ut
il'][color=blue][color=green][color=darkred]
>>> from M2Crypto import RSA
>>> k = RSA.gen_key(512 ,65537)[/color][/color][/color]
............... ..............+ +++++++++++
..............+ +++++++++++[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]


You need mingw versions of libpython23.a, libssl32.a and libeay32.a.

Makefile.mw says:

SSLLIB=c:/pkg/openssl/lib/libssl32.a c:/pkg/openssl/lib/libeay32.a

Whereas, using setup.py, the command line constructed is "-lssleay32
-llibeay32", so you need libssleay32.a and liblibeay32.a. Somewhat
ridiculous looking, yes. ;-). This is because setup.py was written for
VC++; the relevant snippet looks like this:

if os.name == 'nt':
openssl_dir = 'c:\\pkg\\opens sl'
include_dirs = [my_inc, openssl_dir + '/include']
library_dirs = [openssl_dir + '\\lib']
libraries = ['ssleay32', 'libeay32']
#libraries = ['ssleay32_bc', 'libeay32_bc']
extra_compile_a rgs = [ "-DTHREADING" ]

So, alternatively, you can modify the "libraries" line to say "['ssl32',
'eay32']" (I think) to use the default library names assigned to OpenSSL
libraries by mingw. (On Unix they're called libssl.so and libcrypto.so.
Duh!)

Notice "['ssleay_bc', 'libeay_bc']". That's for Borland's free compiler
suite. I haven't tested this in a long while. Going forward, it will be
mingw only.

HTH.


--
Ng Pheng Siong <ngps@netmemeti c.com>

http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
http://sandbox.rulemaker.net/ngps -+- Open Source Python Crypto & SSL

[Carl Waldbieser]

Re: Problem building/using M2Crypto extension module


"Ng Pheng Siong" <ngps@netmemeti c.com> wrote in message[color=blue]
> cd there, import the DLL into the interpreter toplevel:
>
> $ ls -l __m2crypto.so
> -rwxr-xr-x 1 ngps ngps 458146 Sep 28 20:59 __m2crypto.so*
>
> $ python
> Python 2.3 (#3, Sep 15 2003, 15:22:12)
> [GCC 2.95.4 20020320 [FreeBSD]] on freebsd4
> Type "help", "copyright" , "credits" or "license" for more information.[color=green][color=darkred]
> >>> import __m2crypto
> >>>[/color][/color]
>
> Does it work?[/color]
No, it gives me an error each time. Guess there is something wrong with the
..pyd.
[color=blue]
> I've just put up 0.12 snapshot #1. There is a SWIG/Makefile.mw, which is
> for use with mingw. Take a look and adjust paths (e.g., OpenSSL
> include/library files, swig.exe) where necessary. Try it with "make -f
> Makefile.mw".[/color]

That seems to have worked. The .pyd is a little bigger. I copied it into
the site-packages/M2Crypto directory, and I can import M2Crypto now. When I
run the tests, I get:

............... ............... ............... ............... .....EEEEEE
=============== =============== =============== =============== ==========
ERROR: test_cipher_mis match (test_ssl_win.S SLWinClientTest Case)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl.py", line
88, in test_cipher_mis match
pid = self.start_serv er(self.args)
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl_win.py",
line 41, in start_server
None, None, self.startupinf o)
error: (2, 'CreateProcess' , 'The system cannot find the file specified.')

=============== =============== =============== =============== ==========
ERROR: test_cipher_ok (test_ssl_win.S SLWinClientTest Case)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl.py", line
114, in test_cipher_ok
pid = self.start_serv er(self.args)
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl_win.py",
line 41, in start_server
None, None, self.startupinf o)
error: (2, 'CreateProcess' , 'The system cannot find the file specified.')

=============== =============== =============== =============== ==========
ERROR: test_no_such_ci pher (test_ssl_win.S SLWinClientTest Case)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl.py", line
101, in test_no_such_ci pher
pid = self.start_serv er(self.args)
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl_win.py",
line 41, in start_server
None, None, self.startupinf o)
error: (2, 'CreateProcess' , 'The system cannot find the file specified.')

=============== =============== =============== =============== ==========
ERROR: test_server_sim ple (test_ssl_win.S SLWinClientTest Case)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl.py", line
54, in test_server_sim ple
pid = self.start_serv er(self.args)
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl_win.py",
line 41, in start_server
None, None, self.startupinf o)
error: (2, 'CreateProcess' , 'The system cannot find the file specified.')

=============== =============== =============== =============== ==========
ERROR: test_tls1_nok (test_ssl_win.S SLWinClientTest Case)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl.py", line
65, in test_tls1_nok
pid = self.start_serv er(self.args)
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl_win.py",
line 41, in start_server
None, None, self.startupinf o)
error: (2, 'CreateProcess' , 'The system cannot find the file specified.')

=============== =============== =============== =============== ==========
ERROR: test_tls1_ok (test_ssl_win.S SLWinClientTest Case)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl.py", line
77, in test_tls1_ok
pid = self.start_serv er(self.args)
File "C:\Downloads\m 2crypto\m2crypt o-0.12-snap1\tests\tes t_ssl_win.py",
line 41, in start_server
None, None, self.startupinf o)
error: (2, 'CreateProcess' , 'The system cannot find the file specified.')

----------------------------------------------------------------------
Ran 70 tests in 2.530s

FAILED (errors=6)

So it looks like I'm not quite out of the woods yet, but these Python errors
at least seem a little more comprehensible to me. I am still not sure what
was going on with the library. Do you have any idea what I did wrong the
first time, or was it just a lucky guess? I would be interested in knowing
so I can try to avoid that kind of mistake in the future.

Thanks for your help!
Carl Waldbieser

[Ng Pheng Siong]

Re: Problem building/using M2Crypto extension module

According to Carl Waldbieser <waldbie@attglo bal.net>:[color=blue]
> That seems to have worked. The .pyd is a little bigger. I copied it into
> the site-packages/M2Crypto directory, and I can import M2Crypto now. When I
> run the tests, I get:
>
> ............... ............... ............... ............... ....EEEEEE[/color]

These are ok; the errors are in test_ssl_win.py , which says:

Win32 version - requires Mark Hammond's Win32 extensions and openssl.exe
on your PATH.

[color=blue]
> So it looks like I'm not quite out of the woods yet, but these Python errors
> at least seem a little more comprehensible to me. I am still not sure what
> was going on with the library. Do you have any idea what I did wrong the
> first time, or was it just a lucky guess?[/color]

I can't tell, really. As I posted in another message, building for Python
2.3 on Windows works for me.

Anyways, looks like you're good to go.

Cheers.


--
Ng Pheng Siong <ngps@netmemeti c.com>

http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
http://sandbox.rulemaker.net/ngps -+- Open Source Python Crypto & SSL

[Carl Waldbieser]

Re: Problem building/using M2Crypto extension module

Yes, that was the problem-- openssl.exe was not in my PATH. Thanks!

Carl Waldbieser

"Ng Pheng Siong" <ngps@netmemeti c.com> wrote in message > These are ok; the
errors are in test_ssl_win.py , which says:[color=blue]
>
> Win32 version - requires Mark Hammond's Win32 extensions and openssl.exe
> on your PATH.
>[/color]