Code Management

[Jens]

Dear Reader,

I'm writing some modules in Python, and I'm also using unittests. I'm
wondering about some things:

1) Should I put my unittests in a subdirectory? Does the subdirectory
have to be a package?

2) Does the main folder /myproject have to be a package? Should I put
my modules directly under /myproject, or should I create a subfolder,
for example /myproject/modules

Does anyone have any "best practices" as to how to manage your code?

Thanks!

[Sergio Correia]

Re: Code Management

Let's see:

1) Should I put my unittests in a subdirectory? Does the subdirectory
have to be a package?

Sure, to avoid cluttering put the tests in a folder called 'tests'
(and any input the tests require, like mock files, output files used
to test if the output is correct, etc etc, should be put in that
folder or in a subfolder of 'tests'.

I like this setup:

myproject
.../ package1
../../ tests
.../../../ mock_files
.../package2

and so on

If you want to be able to import the tests from python (say, 'from
myproject.tests import test_all'), then yes, you need to make the
tests folder a subpackage (just drop an empty __init__.py file)

2) Does the main folder /myproject have to be a package? Should I put
my modules directly under /myproject, or should I create a subfolder,
for example /myproject/modules

Of course... make /myproject a package. Now, if your modules are kinda
small you can just dump them in /myproject, but if what you are doing
is fairly complex, it's better to move each module to its own folder.

Does anyone have any "best practices" as to how to manage your code?

I want 'best practices' too... anyone?

Sergio


PS: Casey, neat idea with the 'alltests.py'. Btw, I think relative
imports can be helpful:

http://www.python.org/dev/peps/pep-0328/#guido-s-decision

[Ben Finney]

Re: Code Management

Jens <j3nsby@gmail.c omwrites:

On 21 Nov., 04:16, Jens <j3n...@gmail.c omwrote:

On 21 Nov., 01:46, brzr...@gmail.c om wrote:
dummy/
dummy_package/
__init__.py
moduleA.py
tests/
__init__.py
test.py

To avoid confusion, the directory that is the package should be named
as you want the imports to appear; e.g. if you want to 'import
foo.module_a', then name the directory containing 'module_a.py' as
'foo/'.

This often results in::

foo/
setup.py
foo/
__init__.py
module_a.py
test/
__init__.py
test_module_a.p y

That is, the *project* directory (containing all the files) is named
'foo/'; the *package* directory (where all the implementation code is
found) is named 'foo/foo/', and the unit tests are found in the
directory 'foo/test/'.

That's normal, though if it confuses you you might want to rename the
project directory. I'm often doing development on multiple
version-control branches, so each project directory is named for the
branch it contains; within each of those, the same 'foo/' name is used
for the package directory.

I'm using Python 2.5.1. When I'm trying to call a function in
'moduleA' from 'test' it won't work unless I make the 'dummy'
folder a package as well. That's pretty weird. Does
'dummy_package' have to be in my pythonpath or something? How do I
reference moduleA from test?

You should install the package into a place where Python's import path
will find it. Read up on using the standard library 'distutils'
mechanism for this; it involves writing a 'setup.py' file to define
the parameters for installation of your package.

I would like to avoid making 'dummy' into a package as well.

Yes. The top-level directory is used for containing a number of files,
including 'setup.py', that should not be part of the installed
package.

Problem solved. I added 'dummy' to the PYTHONPATH. (Do I really have
to do that for every project I create?) Anyway, it works the way I'd
like it to now.

I hope the above makes it clearer what I prefer for this situation.

--
\ "True greatness is measured by how much freedom you give to |
`\ others, not by how much you can coerce others to do what you |
_o__) want." --Larry Wall |
Ben Finney

[Sergio Correia]

Re: Code Management

As a side note, I find much easier to drop a PTH file than messing
with pythonpath. If you are not familiar with PTH files, what I do is
this

1) Go to "C:\Program Files\Python25\ Lib\site-packages" or whatever is
appropiate in your case.
2) Create a text file, name it something like "MyProjects.PTH " (note
the extension!)
3) in the file, just write the path of the folder that contains all
your projects (in my case, C:/docs/python)

The idea is to keep the main python installation separated from the
modules you are currently developing. Your python installation goes to
"program files/python" or "bin/python", and your personal projects go
somewhere else (usually inside your 'user' folder). This smooths many
things, like working with different versions of a package you are
developing.

On Nov 20, 2007 11:34 PM, Ben Finney <bignose+hate s-spam@benfinney. id.auwrote:

Jens <j3nsby@gmail.c omwrites:
>

On 21 Nov., 04:16, Jens <j3n...@gmail.c omwrote:

On 21 Nov., 01:46, brzr...@gmail.c om wrote:
dummy/
dummy_package/
__init__.py
moduleA.py
tests/
__init__.py
test.py

>
To avoid confusion, the directory that is the package should be named
as you want the imports to appear; e.g. if you want to 'import
foo.module_a', then name the directory containing 'module_a.py' as
'foo/'.
>
This often results in::
>
foo/
setup.py
foo/
__init__.py
module_a.py
test/
__init__.py
test_module_a.p y
>
That is, the *project* directory (containing all the files) is named
'foo/'; the *package* directory (where all the implementation code is
found) is named 'foo/foo/', and the unit tests are found in the
directory 'foo/test/'.
>
That's normal, though if it confuses you you might want to rename the
project directory. I'm often doing development on multiple
version-control branches, so each project directory is named for the
branch it contains; within each of those, the same 'foo/' name is used
for the package directory.
>

I'm using Python 2.5.1. When I'm trying to call a function in
'moduleA' from 'test' it won't work unless I make the 'dummy'
folder a package as well. That's pretty weird. Does
'dummy_package' have to be in my pythonpath or something? How do I
reference moduleA from test?

>
You should install the package into a place where Python's import path
will find it. Read up on using the standard library 'distutils'
mechanism for this; it involves writing a 'setup.py' file to define
the parameters for installation of your package.
>

I would like to avoid making 'dummy' into a package as well.

>
Yes. The top-level directory is used for containing a number of files,
including 'setup.py', that should not be part of the installed
package.
>

Problem solved. I added 'dummy' to the PYTHONPATH. (Do I really have
to do that for every project I create?) Anyway, it works the way I'd
like it to now.

>
I hope the above makes it clearer what I prefer for this situation.
>
--
\ "True greatness is measured by how much freedom you give to |
`\ others, not by how much you can coerce others to do what you |
_o__) want." --Larry Wall |
Ben Finney
>
--

http://mail.python.org/mailman/listinfo/python-list

>

[Jens]

Re: Code Management

On 21 Nov., 05:34, Ben Finney <bignose+hate s-s...@benfinney. id.au>
wrote:

>
I hope the above makes it clearer what I prefer for this situation.
>

It does. You've been most helpful - thanks a lot! I'll bookmark this
thread and keep it under my pillow :-)

[BlueBird]

Re: Code Management

On Nov 21, 7:05 am, "Sergio Correia" <sergio.corr... @gmail.comwrote :

As a side note, I find much easier to drop a PTH file than messing
with pythonpath. If you are not familiar with PTH files, what I do is
this
>
1) Go to "C:\Program Files\Python25\ Lib\site-packages" or whatever is
appropiate in your case.
2) Create a text file, name it something like "MyProjects.PTH " (note
the extension!)
3) in the file, just write the path of the folder that contains all
your projects (in my case, C:/docs/python)
>
The idea is to keep the main python installation separated from the
modules you are currently developing. Your python installation goes to
"program files/python" or "bin/python", and your personal projects go
somewhere else (usually inside your 'user' folder). This smooths many
things, like working with different versions of a package you are
developing.
>

Hi,

If I understand you correctly, you have the following directory
organisation
[Python installation]/site-packages/[MyProject.pth pointing to /home/
user/python-dev/]

And then you do your development in python-dev. But how do you manage
multiple development branches of the same program ?

My directory structure looks like this:
python-dev:
+ prog-branch1/
| + foo
| | + foo.py
| | + tests
| | + test_foo.py
| + bar
| + bar.py
| + tests
| + test_bar.py
+ prog-branch2/
+ foo
| + foo.py
| + tests
| + test_foo.py
+ bar
+ bar.py
+ tests
+ test_bar.py

bar/bar.py needs to import symbols from foo/foo.py . And bar/tests/
test_bar.py needs some symbols from both bar/bar.py and foo/foo.py

I don't understand how having python-dev in the .pth file solves the
problem.

In my case, I make all my imports relative to the root of my project:
bar/bar.py:
from foo.foo import some_foo

bar/tests/test_bar.py
from foo.foo import some_foo
from foo.bar import some_bar

The way I managed to make it work is by extending sys.path but I would
be happy to find a better solution:

bar/bar.py is actually:
import os, sys
sys.path.append ( '..')

from foo.foo import some_foo

and bar/tests/test_bar.py is actually:
import os, sys
sys.path.append ( os.path.join('. .','..') )

from foo.foo import some_foo
from foo.foo import some_bar

What is not nice with this method is that every runnable test script
must extend unconditionally the sys.path . That creates some clutter.

Philippe

[A.T.Hofkamp]

Re: Code Management

On 2007-11-24, BlueBird <phil@freehacke rs.orgwrote:

On Nov 21, 7:05 am, "Sergio Correia" <sergio.corr... @gmail.comwrote :
And then you do your development in python-dev. But how do you manage
multiple development branches of the same program ?

If you are using SVN, you may want to check out 'combinator' by Divmod
(divmod.org). That tool manages SVN branches, and also switches Python when you
switch branches. Very handy.

Albert

[Sergio Correia]

Re: Code Management

Bluebird:

If you are using python 2.5, relative imports are no longer an issue:

http://docs.python.org/whatsnew/pep-328.html

That problem solved, what you sometimes want is to change the version
of your package. I just change the text in the PTH file, to point to
another version, and voilá (no need to move files around or to mess
with the code in the package itself). Probably you can write a script
that changes the PTH file from python itself.

Albert:

Thanks, that looks useful.

[misc@hl.id.au]

Re: Code Management

On Nov 20, 4:09 pm, Jens <j3n...@gmail.c omwrote:

>
1) Should I put my unittests in a subdirectory? Does the subdirectory
have to be a package?

As others have suggested, this is a good way to organise your tests.
To avoid problems with the import path, look at nosetests [1]. This
allows you to run::

nosetests
main_package_na me.subpackage1. test.test1:Test SomeClass.test_ func

to run the test_func function in the TestSomeClass test case.

Or alternatively::

nosetests main_package_na me.subpackage1

to run all tests contained in the main_package_na me.subpackage1
module.

This allows you to remove all of your 'if __name__ ==
"__main__":unit test.main()' code, as well as gets rid of the problem
with relative imports (which others solved using .pth files).

The only downside for me is that running nosetests takes at least half
a second (eg: import nose), compared to making the tests runnable
which can be _very_ quick. For test driven development it's nice to
have tests really really quick.

One gotcha; if you're converting from a 'if __name__ ==
"__main__": ...' system; nosetests by default ignores files that are
executable (so you'll need to chmod -x main_package_na me/subpackage1/
test/test1.py).


[1] nose: a discovery-based unittest extension - (http://
somethingabouto range.com/mrl/projects/nose/).

[BlueBird]

Re: Code Management

On Nov 26, 5:07 pm, "Sergio Correia" <sergio.corr... @gmail.comwrote :

Bluebird:
>
If you are using python 2.5, relative imports are no longer an
issue:http://docs.python.org/whatsnew/pep-328.html

It does not solve my problem, or I missed something:

User@Phil_vaio /cygdrive/d/work/work/vy-dev/foo $ find . -name '*.py'
../bar/bar.py
../bar/tests/test_bar.py
../bar/tests/test_bar1.py
../bar/tests/__init__.py
../bar/__init__.py
../foo/foo.py
../foo/tests/__init__.py
../foo/__init__.py
../__init__.py

d:\work\work\vy-dev\foo>C:\Pyth on25\python.exe bar/tests/test_bar.py
d:\work\work\vy-dev\foo
['d:\\work\\work \\vy-dev\\foo\\bar\\ tests', 'C:\\Python25\\ lib\\site-
packages\\h
gsvn-0.1.4-py2.5.egg', 'C:\\WINDOWS\\s ystem32\\python 25.zip', 'C:\
\Python25\\DLL
s', 'C:\\Python25\\ lib', 'C:\\Python25\\ lib\\plat-win', 'C:\\Python25\
\lib\\lib-
tk', 'C:\\Python25', 'C:\\Python25\\ lib\\site-packages', 'C:\\Python25\
\lib\\sit
e-packages\\win32 ', 'C:\\Python25\\ lib\\site-packages\\win32 \\lib', 'C:
\\Python2
5\\lib\\site-packages\\Pytho nwin', 'C:\\Python25\\ lib\\site-packages\
\wx-2.8-msw
-ansi']
Traceback (most recent call last):
File "bar/tests/test_bar.py", line 8, in <module>
from ..bar import bar_f
ValueError: Attempted relative import in non-package

[BlueBird]

Re: Code Management

On Dec 2, 4:27 pm, BlueBird <p...@freehacke rs.orgwrote:

On Nov 26, 5:07 pm, "Sergio Correia" <sergio.corr... @gmail.comwrote :
>

Bluebird:

>

If you are using python 2.5, relative imports are no longer an
issue:http://docs.python.org/whatsnew/pep-328.html

>
It does not solve my problem, or I missed something:
>

Any idea what could be wrong in my code (in the previous message). Or
did I misunderstand relative import seriously ?

Philippe

[Marc Christiansen]

Re: Code Management

BlueBird <phil@freehacke rs.orgwrote:

On Dec 2, 4:27 pm, BlueBird <p...@freehacke rs.orgwrote:

>On Nov 26, 5:07 pm, "Sergio Correia" <sergio.corr... @gmail.comwrote :
>>

>Bluebird:

>>

If you are using python 2.5, relative imports are no longer an
issue:http://docs.python.org/whatsnew/pep-328.html

>>
>It does not solve my problem, or I missed something:
>>

>
Any idea what could be wrong in my code (in the previous message). Or
did I misunderstand relative import seriously ?

Does this give you an idea?

0:tolot:/tmpmkdir t1
0:tolot:/tmptouch t1/__init__.py
0:tolot:/tmpecho print __name__ >t1/test.py
0:tolot:/tmppython t1/test.py
__main__
0:tolot:/tmppython -c "import t1.test"
t1.test

Marc