Module Structure/Import Design Problem

[Rafe]

Hi,

I am in a situation where I feel I am being forced to abandon a clean
module structure in favor of a large single module. If anyone can save
my sanity here I would be forever grateful.

My problem is that classes in several modules share a common base
class which needs to implement a factory method to return instances of
these same classes.

An example to help illustrate what I mean:
Lets say I have the following modules with the listed classes:
- baselib.py with BaseClass
- types.py with TypeA, ...
- special.py with SpecialTypeA, ...

Which would be used a bit like this:

>>type_a = any_type_instan ce.get_type("Ty peA")
>>special_typ e = type_a.get_type ("SpecialTypeA" )

Again, I can get around this by dumping everything in to one module,
but it muddies the organization of the package a bit. This seems like
a problem that would come up a lot. Are there any design paradigms I
can apply here?

Cheers

- Rafe

[Arnaud Delobelle]

Re: Module Structure/Import Design Problem

Rafe <rafesacks@gmai l.comwrites:

Hi,
>
I am in a situation where I feel I am being forced to abandon a clean
module structure in favor of a large single module. If anyone can save
my sanity here I would be forever grateful.
>
My problem is that classes in several modules share a common base
class which needs to implement a factory method to return instances of
these same classes.
>
An example to help illustrate what I mean:
Lets say I have the following modules with the listed classes:
- baselib.py with BaseClass
- types.py with TypeA, ...
- special.py with SpecialTypeA, ...
>
Which would be used a bit like this:

>>>type_a = any_type_instan ce.get_type("Ty peA")
>>>special_ty pe = type_a.get_type ("SpecialTypeA" )

>
>
Again, I can get around this by dumping everything in to one module,
but it muddies the organization of the package a bit. This seems like
a problem that would come up a lot. Are there any design paradigms I
can apply here?

It's not very clear what your problem is. I guess your factory
functions are defined in baselib.py whereas types.py and special.py
import baselib, therefore you don't know how to make the factory
function aware of the types defined in special.py and types.py.

You can use cyclic import in many cases.

Or (better IMHO) you can make types register themselves with the factory
function (in which case it would have some state so it would make more
sense to make it a factory object).

--
Arnaud

[Rafe]

Re: Module Structure/Import Design Problem

On Nov 20, 2:06 pm, Arnaud Delobelle <arno...@google mail.comwrote:

Rafe <rafesa...@gmai l.comwrites:

Hi,

>

I am in a situation where I feel I am being forced to abandon a clean
module structure in favor of a large single module. If anyone can save
my sanity here I would be forever grateful.

>

My problem is that classes in several modules share a common base
class which needs to implement a factory method to return instances of
these same classes.

>

An example to help illustrate what I mean:
Lets say I have the following modules with the listed classes:
 - baselib.py   with  BaseClass
 - types.py   with  TypeA, ...
 - special.py   with  SpecialTypeA, ...

>

Which would be used a bit like this:

>>type_a = any_type_instan ce.get_type("Ty peA")
>>special_typ e = type_a.get_type ("SpecialTypeA" )

>

Again, I can get around this by dumping everything in to one module,
but it muddies the organization of the package a bit. This seems like
a problem that would come up a lot. Are there any design paradigms I
can apply here?

>
It's not very clear what your problem is.  I guess your factory
functions are defined in baselib.py whereas types.py and special.py
import baselib, therefore you don't know how to make the factory
function aware of the types defined in special.py and types.py.
>
You can use cyclic import in many cases.
>
Or (better IMHO) you can make types register themselves with the factory
function (in which case it would have some state so it would make more
sense to make it a factory object).
>
--
Arnaud

hi Arnaud,

You got my problem right, sorry it wasn't more clear.

Can you elaborate on what you mean by 'register' with the factory
function?

Also...holy cr@p, I got a clean import working! I swear I tried that
before with unhappy results. I'll carefully try this in my real code.

Is this the right way to impliment the imports?....

baselib.py
[1] class BaseClass(objec t):
[2] def factory(self):
[3] import typelib # <-- import inside function
[4] return typelib.TypeA()

typelib.py
[1] import baselib # <-- module level import
[2]
[3] class TypeA(baselib.B aseClass):
[4] def __init__(self):
[5] print "TypeA : __init__()"

>>import typelib
>>type = typelib.TypeA()

TypeA : __init__()

>>another_typ e = type.factory()

TypeA : __init__()

>>another_typ e

<typelib.Type A object at 0x00B45F10>


I am curious (not directed at Arnaud), why not have an 'include'-like
import for special cases in python (or do I not understand includes
either?)

Thanks!

- Rafe

[Stef Mientki]

Re: Module Structure/Import Design Problem

Rafe wrote:

Hi,
>
I am in a situation where I feel I am being forced to abandon a clean
module structure in favor of a large single module. If anyone can save
my sanity here I would be forever grateful.
>
My problem is that classes in several modules share a common base
class which needs to implement a factory method to return instances of
these same classes.
>
An example to help illustrate what I mean:
Lets say I have the following modules with the listed classes:
- baselib.py with BaseClass
- types.py with TypeA, ...
- special.py with SpecialTypeA, ...
>
Which would be used a bit like this:
>

>>>type_a = any_type_instan ce.get_type("Ty peA")
>>>special_ty pe = type_a.get_type ("SpecialTypeA" )
>>>>

>
>
Again, I can get around this by dumping everything in to one module,
but it muddies the organization of the package a bit. This seems like
a problem that would come up a lot. Are there any design paradigms I
can apply here?
>
>

I'm not an expert, I even don't fully understand your problem,
but having struggled with imports in the past,
I've a solution now, which seems to work quit well.

cheers,
Stef

Cheers
>
- Rafe
>
>
--

http://mail.python.org/mailman/listinfo/python-list

>

[Steve Holden]

Re: Module Structure/Import Design Problem

Rafe wrote:

Hi,
>
I am in a situation where I feel I am being forced to abandon a clean
module structure in favor of a large single module. If anyone can save
my sanity here I would be forever grateful.
>
My problem is that classes in several modules share a common base
class which needs to implement a factory method to return instances of
these same classes.
>
An example to help illustrate what I mean:
Lets say I have the following modules with the listed classes:
- baselib.py with BaseClass
- types.py with TypeA, ...
- special.py with SpecialTypeA, ...
>
Which would be used a bit like this:

>>>type_a = any_type_instan ce.get_type("Ty peA")
>>>special_ty pe = type_a.get_type ("SpecialTypeA" )

>
>
Again, I can get around this by dumping everything in to one module,
but it muddies the organization of the package a bit. This seems like
a problem that would come up a lot. Are there any design paradigms I
can apply here?
>

Well a simple way to do this is to observe that even when a base class's
method is inherited by an instance of a subclass, when the method is
called the type of "self" is the subclass. And you can call the
subclass's type to create an instance. Perhaps the following code would
make it more obvious:

$ cat baseclass.py
class Base(object):
def factory(self, arg):
return type(self)(arg)

sholden@lifeboy /c/Users/sholden/Projects/Python
$ cat subclass.py
from baseclass import Base

class sub(Base):
def __init__(self, arg):
print "Creating a sub with arg", arg

s = sub("Manual")

thing = s.factory("Auto ")
print type(thing)


sholden@lifeboy /c/Users/sholden/Projects/Python
$ python subclass.py
Creating a sub with arg Manual
Creating a sub with arg Auto
<class '__main__.sub'>

Hope this helps.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

[Steve Holden]

Re: Module Structure/Import Design Problem

Stef Mientki wrote:

Rafe wrote:

>Hi,
>>
>I am in a situation where I feel I am being forced to abandon a clean
>module structure in favor of a large single module. If anyone can save
>my sanity here I would be forever grateful.
>>
>My problem is that classes in several modules share a common base
>class which needs to implement a factory method to return instances of
>these same classes.
>>
>An example to help illustrate what I mean:
>Lets say I have the following modules with the listed classes:
> - baselib.py with BaseClass
> - types.py with TypeA, ...
> - special.py with SpecialTypeA, ...
>>
>Which would be used a bit like this:
>>

>>>>type_a = any_type_instan ce.get_type("Ty peA")
>>>>special_typ e = type_a.get_type ("SpecialTypeA" )
>>>>>

>>
>>
>Again, I can get around this by dumping everything in to one module,
>but it muddies the organization of the package a bit. This seems like
>a problem that would come up a lot. Are there any design paradigms I
>can apply here?
>>
>>

I'm not an expert, I even don't fully understand your problem,
but having struggled with imports in the past,
I've a solution now, which seems to work quit well.
>

That's not very helpful, is it? Were you planning to keep the solution
secret?

regards
steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

[Stef Mientki]

Re: Module Structure/Import Design Problem

>>>
>>>

>I'm not an expert, I even don't fully understand your problem,
>but having struggled with imports in the past,
>I've a solution now, which seems to work quit well.
>>
>>

That's not very helpful, is it? Were you planning to keep the solution
secret?
>

sorry slip of the keyboard ;-)

http://mientki.ruhosting.nl/data_www/pylab_works/pw_importing.html

cheers,
Stef

[Rafe]

Re: Module Structure/Import Design Problem

On Nov 21, 1:39 am, Steve Holden <st...@holdenwe b.comwrote:

Rafe wrote:

Hi,

>

I am in a situation where I feel I am being forced to abandon a clean
module structure in favor of a large single module. If anyone can save
my sanity here I would be forever grateful.

>

My problem is that classes in several modules share a common base
class which needs to implement a factory method to return instances of
these same classes.

>

An example to help illustrate what I mean:
Lets say I have the following modules with the listed classes:
 - baselib.py   with  BaseClass
 - types.py   with  TypeA, ...
 - special.py   with  SpecialTypeA, ...

>

Which would be used a bit like this:

>>type_a = any_type_instan ce.get_type("Ty peA")
>>special_typ e = type_a.get_type ("SpecialTypeA" )

>

Again, I can get around this by dumping everything in to one module,
but it muddies the organization of the package a bit. This seems like
a problem that would come up a lot. Are there any design paradigms I
can apply here?

>
Well a simple way to do this is to observe that even when a base class's
method is inherited by an instance of a subclass, when the method is
called the type of "self" is the subclass. And you can call the
subclass's type to create an instance. Perhaps the following  code would
make it more obvious:
>
$ cat baseclass.py
class Base(object):
    def factory(self, arg):
        return type(self)(arg)
>
sholden@lifeboy /c/Users/sholden/Projects/Python
$ cat subclass.py
from baseclass import Base
>
class sub(Base):
  def __init__(self, arg):
    print "Creating a sub with arg", arg
>
s = sub("Manual")
>
thing = s.factory("Auto ")
print type(thing)
>
sholden@lifeboy /c/Users/sholden/Projects/Python
$ python subclass.py
Creating a sub with arg Manual
Creating a sub with arg Auto
<class '__main__.sub'>
>
Hope this helps.
>
regards
 Steve
--
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/

Hi Steve,

Correct me if I have this wrong, but the problem with your solution is
that it only creates a new instance of the same class, type(self),
while I need to return any number of different possibilities.

I thought about getting the module from self...

>>class base(object):
>> def factory(self, type):
>> module = sys.modules[self.__class__. __module__]
>> return getattr(module, type)

....but my baseclass is used from several modules so this would be
inaccurate for me (the factory method only uses my 'types' module, so
a hard import works)

I'm still wondering what Arnaud meant by "make types register
themselves with the factory function"

- Rafe

[Rafe]

Re: Module Structure/Import Design Problem

On Nov 21, 2:36 am, Stef Mientki <stef.mien...@g mail.comwrote:

I'm not an expert, I even don't fully understand your problem,
but having struggled with imports in the past,
I've a solution now, which seems to work quit well.

>

That's not very helpful, is it? Were you planning to keep the solution
secret?

>
sorry slip of the keyboard ;-)http://mientki.ruhosting.nl/data_www...importing.html
cheers,
Stef

I really don't understand what you are trying to accomplish in your
article.

I strongly disagree with this statement...
"A second demand is that every module should be able to act as a main
file by running it's main section."
....I am finding the best programs have only one entry point or
interface (though some libraries can be useful from outside the
package.) Being able to run any file in a package seems like it
creates a confusing user/developer experience. What kind of problem
makes this solution applicable?

Next, you say...
"...recursi ve searches for all subdirectories and adds them to the
Python Path."
....it seems like you add every module in your packages directly to the
sys.path. Doesn't this destroy the package name-spaces? For example, I
have a module called 'types' in my package if I add that to the python
path, 'import types' still returns the built-in 'types' module.
Wouldn't this collision be confusing? Regardless, isn't putting the
package in the right place enough? Please explain.


Cheers,

- Rafe

[Steve Holden]

Re: Module Structure/Import Design Problem

Rafe wrote:

On Nov 21, 1:39 am, Steve Holden <st...@holdenwe b.comwrote:

>Rafe wrote:

>>Hi,
>>I am in a situation where I feel I am being forced to abandon a clean
>>module structure in favor of a large single module. If anyone can save
>>my sanity here I would be forever grateful.
>>My problem is that classes in several modules share a common base
>>class which needs to implement a factory method to return instances of
>>these same classes.
>>An example to help illustrate what I mean:
>>Lets say I have the following modules with the listed classes:
>> - baselib.py with BaseClass
>> - types.py with TypeA, ...
>> - special.py with SpecialTypeA, ...
>>Which would be used a bit like this:
>>>>>type_a = any_type_instan ce.get_type("Ty peA")
>>>>>special_ty pe = type_a.get_type ("SpecialTypeA" )
>>Again, I can get around this by dumping everything in to one module,
>>but it muddies the organization of the package a bit. This seems like
>>a problem that would come up a lot. Are there any design paradigms I
>>can apply here?

>Well a simple way to do this is to observe that even when a base class's
>method is inherited by an instance of a subclass, when the method is
>called the type of "self" is the subclass. And you can call the
>subclass's type to create an instance. Perhaps the following code would
>make it more obvious:
>>
>$ cat baseclass.py
>class Base(object):
> def factory(self, arg):
> return type(self)(arg)
>>
>sholden@lifebo y /c/Users/sholden/Projects/Python
>$ cat subclass.py
>from baseclass import Base
>>
>class sub(Base):
> def __init__(self, arg):
> print "Creating a sub with arg", arg
>>
>s = sub("Manual")
>>
>thing = s.factory("Auto ")
>print type(thing)
>>
>sholden@lifebo y /c/Users/sholden/Projects/Python
>$ python subclass.py
>Creating a sub with arg Manual
>Creating a sub with arg Auto
><class '__main__.sub'>
>>
>Hope this helps.
>>
>regards
> Steve
>--
>Steve Holden +1 571 484 6266 +1 800 494 3119
>Holden Web LLC http://www.holdenweb.com/

>
Hi Steve,
>
Correct me if I have this wrong, but the problem with your solution is
that it only creates a new instance of the same class, type(self),
while I need to return any number of different possibilities.
>

In that case you need to pass the type of the new instance you require
as an argument to the factory method, I guess, or something similar. My
example has each subclass returning instances of the subclass that was
used to call the factory method, yes.

If A instances need to be able to get B instances and B instances need
to be able to create B instances then you do indeed have a tricky
problem when it comes to separating your classes into different
modules,. But it's not insoluble!

I thought about getting the module from self...

>>>class base(object):
>>> def factory(self, type):
>>> module = sys.modules[self.__class__. __module__]
>>> return getattr(module, type)

>
...but my baseclass is used from several modules so this would be
inaccurate for me (the factory method only uses my 'types' module, so
a hard import works)
>

I am not sure yet if I understand the requirement properly. It seems
from the above code that you always want to create instances of a named
type defined in the same module as the subclass?

I'm still wondering what Arnaud meant by "make types register
themselves with the factory function"
>

You would have to ask him, but I suspect he means having each subtype
call a method of the base type to add itself to some dictionary so you
can call the factory method with a key that will tell it which subtype
to produce. This is a fairly sensible way to separate definitions form
references.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

[Gabriel Genellina]

Re: Module Structure/Import Design Problem

En Thu, 20 Nov 2008 17:36:11 -0200, Stef Mientki <stef.mientki@g mail.com>
escribió:

>>I'm not an expert, I even don't fully understand your problem,
>>but having struggled with imports in the past,
>>I've a solution now, which seems to work quit well.
>>>

>That's not very helpful, is it? Were you planning to keep the solution
>secret?
>>

sorry slip of the keyboard ;-)
http://mientki.ruhosting.nl/data_www...importing.html

May I reiterate my criticism to your "solution" posted last week?
I don't think extending sys.path to include every directory under your
project is a good idea. Basically, you're flattening the directory layout,
removing any structure, like it was before Python 1.5 added package
support.
It is like dumping all your modules in a single directory, with no
hierarchy and lots of name conflicts.
Worse: because your proposal makes the same file reachable under many
different names, the *same* source module will generate *different* module
objects stored as *different* entries in sys.modules. Globals don't work
anymore, subclasses aren't subclasses... lots of problems.
Relative imports (PEP328 [1]) were introduced -among other things- as an
attempt to avoid such problems. You're going in the opposite direction.
Please stop doing that - or at least keep us informed of where you work!

[1] http://www.python.org/dev/peps/pep-0328

--
Gabriel Genellina

[Stef Mientki]

Re: Module Structure/Import Design Problem

Rafe wrote:

On Nov 21, 2:36 am, Stef Mientki <stef.mien...@g mail.comwrote:
>

>>>I'm not an expert, I even don't fully understand your problem,
>>>but having struggled with imports in the past,
>>>I've a solution now, which seems to work quit well.
>>>>
>>That's not very helpful, is it? Were you planning to keep the solution
>>secret?
>>>

>sorry slip of the keyboard ;-)http://mientki.ruhosting.nl/data_www...importing.html
>cheers,
>Stef
>>

>
I really don't understand what you are trying to accomplish in your
article.
>
I strongly disagree with this statement...
"A second demand is that every module should be able to act as a main
file by running it's main section."
...I am finding the best programs have only one entry point or
interface (though some libraries can be useful from outside the
package.)

At first the program already basically consists of 3 different user
interfaces, MatLab-like, LabView-like, Punthoofd-like.
Then it maybe a matter of taste, but for the moment I've planned to have
3 to 5 different IDEs (besides the main programs), but that might change
when they are all finished.
It might be a matter of taste, but I think you'll be better of by an
optimal dedicated IDE, instead of something like Word ( I estimate that
I, as many others, use less than 1% of the items, so you can imagine how
much time we spent on searching the right item)

Being able to run any file in a package seems like it
creates a confusing user/developer experience. What kind of problem
makes this solution applicable?
>
>

Testing ...
.... by others on other operating systems
.... in combination with other Python versions
.... in combination with other Library versions (the program interacts
heavily with wxPython, Numpy, Scipy, MatPlotLib, VPython, PyGame, PIL,
Rpyc, winpdb, wmi , LXML., ConfigObj, Nucular, ...)
.... automatically testing after modifications

Next, you say...
"...recursi ve searches for all subdirectories and adds them to the
Python Path."
...it seems like you add every module in your packages directly to the
sys.path. Doesn't this destroy the package name-spaces? For example, I
have a module called 'types' in my package if I add that to the python
path, 'import types' still returns the built-in 'types' module.
Wouldn't this collision be confusing? Regardless, isn't putting the
package in the right place enough? Please explain.
>

Well I've to confess, I'm not a programmer,
And if I can program, well, I leave that up to you,
and is not very relevant, because I know I can make useful programs ;-)
But I've to admit, I don't understand packages.
I've never experienced any collision,
but then I probably always use much longer, self explaining names than
the build-ins.

This my directory structure ( no module names) , which only contains the
program and its libraries.
A few paths are 1 or 2 levels deeper.
The " lang" contains internationaliz ation strings in normal python
files (another 300 files per language ;-)
Now in my opinion, each module in each path should be able to reach any
other module in this directory structure.
So other / better / simpler solutions are welcome.

cheers,
Stef

main_path
|__ lang
|__ sub_path1
|__lang
|__sub_sub_path _1a
|__lang
|__sub_sub_path _1b
|__lang
|__sub_sub_path _1c
|__lang
|__ sub_path2
|__lang
|__sub_sub_path _2a
|__lang
|__sub_sub_path _2b
|__lang
....
|__ sub_path8

>
Cheers,
>
- Rafe
--

http://mail.python.org/mailman/listinfo/python-list

>

[Stef Mientki]

Re: Module Structure/Import Design Problem

Gabriel Genellina wrote:

En Thu, 20 Nov 2008 17:36:11 -0200, Stef Mientki
<stef.mientki@g mail.comescribi ó:
>

>>>I'm not an expert, I even don't fully understand your problem,
>>>but having struggled with imports in the past,
>>>I've a solution now, which seems to work quit well.
>>>>
>>That's not very helpful, is it? Were you planning to keep the solution
>>secret?
>>>

>sorry slip of the keyboard ;-)
>http://mientki.ruhosting.nl/data_www...importing.html

>
May I reiterate my criticism to your "solution" posted last week?

You're welcome,
if there are better (even more important simpler) solutions, I'm in.

I don't think extending sys.path to include every directory under your
project is a good idea. Basically, you're flattening the directory
layout, removing any structure, like it was before Python 1.5 added
package support.
It is like dumping all your modules in a single directory, with no
hierarchy and lots of name conflicts.

Well I started at 2.4 and now arrived at 2.5, so I don't know what it
was at 1.5.
And I'm probably going to stay at 2.5 for a long time, upgrading is one
of the few (maybe the only) disadvantages of Python ;-)
For Python, I guess you're right, all files are in one flat directory, ...
.... but it's just 1 program, so what's the objection ?
For the program itself, which is highly dynamic, it can find sets of
python-files in certain subpaths,
so that's quit ordered I think ?

Worse: because your proposal makes the same file reachable under many
different names, the *same* source module will generate *different*
module objects stored as *different* entries in sys.modules. Globals
don't work anymore, subclasses aren't subclasses... lots of problems.

Could you explain this a little more ...
.... I'm just adding every subpath to the Pythonpath,
so in my ignorant believe, every module is imported the same way, but I
might be wrong.
" Globals not working" , a good program doesn't have any globals ( said
a non-programmer ;-)
Subclasses aren't subclasses ? ( I always derive subclasses in the
module itself, or in the same directory as the parentclass)

Relative imports (PEP328 [1]) were introduced -among other things- as
an attempt to avoid such problems. You're going in the opposite
direction. Please stop doing that - or at least keep us informed of
where you work!
>
[1] http://www.python.org/dev/peps/pep-0328
>

Sorry I don't understand all that pep-talk (I'm not a programmer ;-)
I read it starts with " For the second problem, it is proposed that all
import statements be absolute by default (searching sys.path only) with
special syntax (leading dots) for accessing package-relative imports."
I think that's exactly what I'm doing: absolute imports.

Please explain in simple language what I'm all doing wrong,
or how I get a better solution.

thanks,
Stef

[Steve Holden]

Re: Module Structure/Import Design Problem

Stef Mientki wrote:

Gabriel Genellina wrote:

[...]

Sorry I don't understand all that pep-talk (I'm not a programmer ;-)

And I'm not a plumber. The difference between us is that I don't write
blogs telling people how to lay out and connect their pipework.

regards
Steve

PS: Q: What's the difference between God and a doctor?
A: God doesn't think she's a doctor
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/