Project organization and import

[sjdevnull@yahoo.com]

Re: Project organization and import

On Mar 6, 12:49 pm, "Martin Unsal" <martinun...@gm ail.comwrote:

On Mar 6, 9:19 am, "Chris Mellon" <arka...@gmail. comwrote:
>

You do? Or do you only have trouble because you don't like using "from
foo import Foo" because you need to do more work to reload such an
import?

>
More work, like rewriting __import__ and reload??? :)
>
There's a point where you should blame the language, not the
programmer. Are you saying I'm lazy just because I don't want to mess
with __import__?
>

What makes you think that the exposed namespace has to be isomorphic
with the filesystem?

>
I don't; you do!
>
I was clearly talking about files and you assumed I was talking about
namespace. That's Pythonic thinking... and I don't mean that in a good
way!
>

If you want to break a module into multiple packages and then stick
the files that make up the package in bizarre spots all over the
filesystem, can you give a reason why?

>
Because I have written a project with 50,000 lines of Python and I'm
trying to organize it in such a way that it'll scale up cleanly by
another order of magnitude. Because I've worked on projects with
millions of lines of code and I know about how such things are
organized. It's funny, I'm a newbie to Python but it seems like I'm
one of the only people here thinking about it as a large scale
development language rather than a scripting language.
>
Martin

I'm still not clear on what your problem is or why you don't like
"from foo import bar". FWIW our current project is about 330,000
lines of Python code. I do a ton of work in the interpreter--I'll
often edit code and then send a few lines over to the interpreter to
be executed. For simple changes, reload() works fine; for more
complex cases we have a reset() function to clear out most of the
namespace and re-initialize. I don't really see how reload could be
expected to guess, in general, what we'd want reloaded and what we'd
want kept, so I have a hard time thinking of it as a language problem.

[sjdevnull@yahoo.com]

Re: Project organization and import

On Mar 6, 4:58 pm, Ben Finney <bignose+hate s-s...@benfinney. id.au>
wrote:

"Martin Unsal" <martinun...@gm ail.comwrites:

I think you should be asking yourselves, "Did we all abandon reload()
because it is actually an inferior workflow, or just because it's
totally broken in Python?"

>
I never "abandoned reload()", because it never even occurred to me to
use the interpreter for developing the code that I know is going to
end up in a file anyway. That's what my text editor is for.

It's most useful for debugging for me; I'll instantiate the objects of
a known bad test case, poke around, maybe put some more debugging code
into one of my classes and re-instantiate only those objects (but keep
the rest of the test objects as-is).

Even there I find that I'd rather use a scratch file in an editor to
set up the test cases and send a specified region to the interpreter
for the most part, only actually typing in the interpreter when I'm
poking at an object. I'll often wind up wanting to pull part of the
test case out either to go into the production code or to set up a
permanent unit test.

Once I figure out what's going on, the production code definitely gets
edited in the text editor.

Even though I use the interactive interpreter every day, though, I
haven't noticed reload being a major issue.

[Terry Hancock]

Re: Project organization and import

Martin Unsal wrote:

I'm using Python for what is becoming a sizeable project and I'm
already running into problems organizing code and importing packages.
I feel like the Python package system, in particular the isomorphism
between filesystem and namespace, doesn't seem very well suited for
big projects.

I've never worked on what you would call a "big project", but I *am*
kind of a neat-freak/control-freak about file organization of code, so I
have tinkered with the structure of source trees in Python quite a bit.

If you want to explode a module into a lot of smaller files, you create
a package. I find that this usually works best like this (this is what
the filesystem looks like):

package_name/
package_pre.py - contains globals for the package
component_a.py - a useful-sized collection of functionality
component_b.py - another
component_c.py - another
package_post.py - stuff that relies on the prior stuff
__init__.py - or you can put the "post" stuff here

Then __init__.py contains something like:

from package_pre import *
from component_a import *
from component_b import *
from component_c import *
from package_post import *

or you can explicitly load what you need:

from package_pre import *
from component_a import A, A1, A2
from component_a import A3 as A5
from component_b import B, B1
from component_c import C, C2, C5
from package_post import *

if you want to keep the namespace cleaner.

Also, instead of just dropping things into the module's global
namespace, use an named namespace, such as a class, or use the
"package_pr e" in the example above. That helps to keep things separable.

IOW, you can use __init__.py to set up the package's namespace anyway
you want, breaking the actual code up into just about as many files as
you like (I also don't like reading long source files -- I find it
easier to browse directories than source files, even with outlining
extensions. It's rare for me to have more than 2-3 classes per file).

Of course, if you *really* want your namespace to be *completely*
different from the filesystem, then there's no actual reason that all of
these files have to be in the same directory. You can use Python's
relative import (standard in Python 2.5+, available using __future__ in
2.4, IIRC) to make this easier. There was an obnoxious hack used in Zope
which used code to extract the "package_pa th" and then prepend that to
get absolute import locations which was necessary in earlier versions --
but I can't recommend that, just use the newer version of Python.

So, you could do evil things like this in __init__.py:

from .other_package. fiddly_bit import dunsel

(i.e. grab a module from a neighboring package)

Of course, I really can't recommend that either. Python will happily do
it, but it's a great way to shoot yourself in the foot in terms of
keeping your code organized!

The only exception to that is that I often have a "util" or "utility"
package which has a collection of little extras I find useful throughout
my project.

As for relying heavily on reload(), it isn't that great of a feature for
debugging large projects. Any code of sufficient size to make reload()
problematic, though, needs formal unit testing, anyway. The cheapest and
easiest unit test method is doctests (IMHO), so you ought to give those
a try -- I think you'll like the easy relationship those have to working
in the interactive interpreter: just walk your objects through their
paces in the interpreter, then cut-and-paste.

What reload() and the interactive interpreter is good for is
experimentation , not development.

If you need huge amounts of code to be loaded to be able to do any
useful experiments with the modules you are writing, then your code is
too tightly coupled to begin with. Try to solve that by using something
like "mock objects" to replace the full blown implementations of objects
you need for testing. I've never formally used any of the "mock"
packages, but I have done a number of tests using objects which are
dumbed-down versions of objects which are really supposed to be provided
from another module -- but I wanted to test the two separately (which is
essentially creating my own mock objects from scratch).

HTH,
Terry

--
Terry Hancock (hancock@Anansi Spaceworks.com)
Anansi Spaceworks http://www.AnansiSpaceworks.com

[Michele Simionato]

Re: Project organization and import

On Mar 5, 1:21 am, "Martin Unsal" <martinun...@gm ail.comwrote:

2) Importing and reloading. I want to be able to reload changes
without exiting the interpreter.

What about this?

$ cat reload_obj.py
"""
Reload a function or a class from the filesystem.

For instance, suppose you have a module

$ cat mymodule.py
def f():
print 'version 1 of function f'

Suppose you are testing the function from the interactive interpreter:

>>from mymodule import f
>>f()

version 1 of function f

Then suppose you edit mymodule.py:

$ cat mymodule.py
def f():
print 'version 2 of function f'

You can see the changes in the interactive interpreter simply by doing

>>f = reload_obj(f)
>>f()

version 2 of function f
"""

import inspect

def reload_obj(obj) :
assert inspect.isfunct ion(obj) or inspect.isclass (obj)
mod = __import__(obj. __module__)
reload(mod)
return getattr(mod, obj.__name__)

Pretty simple, isn't it?

The issue is that if you have other objects dependending on the
previous version
of the function/class, they will keep depending on the previous
version, not on
the reloaded version, but you cannot pretende miracles from reload! ;)

You can also look at Michael Hudson's recipe

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

for a clever approach to automatic reloading.

Michele Simionato

[garylinux@gmail.com]

Re: Project organization and import

package_name/

package_pre.py - contains globals for the package
component_a.py - a useful-sized collection of functionality
component_b.py - another
component_c.py - another
package_post.py - stuff that relies on the prior stuff
__init__.py - or you can put the "post" stuff here
>
Then __init__.py contains something like:
>
from package_pre import *
from component_a import *
from component_b import *
from component_c import *
from package_post import *
>
Anansi Spaceworkshttp://www.AnansiSpace works.com

Thank you! That is by far the clearest I have ever seen that
explained.
I saved it and Sent it on to a friend that is learning python.