Best practices for dynamically loading plugins at startup

[Christoph Haas]

Dear coders...

I'm working on an application that is supposed to support "plugins".
The idea is to use the plugins as packages like this:

Plugins/
__init__.py
Plugin1.py
Plugin2.py
Plugin3.py

When the application starts up I want to have these modules loaded
dynamically. Users can put their own plugin modules into the
Plugins/ directory and the application should know about it.

Since I don't know which plugins have been put into that directory
I cannot just "import Plugin1, Plugin2, Plugin3" in the "__init__.p y".
So I need to find out the *.py there and load them during startup.
I could do that with a "walk" over that directory.

Each plugin is supposed to be a class derived from a general
"Plugin" superclass. I just don't know how to 'register' every
plugin. The main application needs to know which plugin classes
there are. On IRC I was recommended walking through all objects
and finding out if the class is a subclass of "Plugin". Another
person recommended using metaclasses to automatically register
the plugin in a global list.

Since I have only little real-life Python knowledge I wonder what the
best practice for this kind of problem is.

I looked at the "supybot" IRC bot to get an idea how plugins are handled
there. Unfortunately it was still a bit over my (python) head.

Regards
Christoph
--
~
~
~
".signature " [Modified] 3 lines --100%-- 3,41 All

[Jarek Zgoda]

Re: Best practices for dynamically loading plugins at startup

Christoph Haas napisa³(a):
[color=blue]
> Since I don't know which plugins have been put into that directory
> I cannot just "import Plugin1, Plugin2, Plugin3" in the "__init__.p y".
> So I need to find out the *.py there and load them during startup.
> I could do that with a "walk" over that directory.[/color]

See entry for __import__ at

http://www.python.org/doc/2.3/lib/built-in-funcs.html

--
Jarek Zgoda

http://jpa.berlios.de/

[Jeff Schwab]

Re: Best practices for dynamically loading plugins at startup

Christoph Haas wrote:[color=blue]
> Dear coders...
>
> I'm working on an application that is supposed to support "plugins".
> The idea is to use the plugins as packages like this:
>
> Plugins/
> __init__.py
> Plugin1.py
> Plugin2.py
> Plugin3.py
>
> When the application starts up I want to have these modules loaded
> dynamically. Users can put their own plugin modules into the
> Plugins/ directory and the application should know about it.
>
> Since I don't know which plugins have been put into that directory
> I cannot just "import Plugin1, Plugin2, Plugin3" in the "__init__.p y".
> So I need to find out the *.py there and load them during startup.
> I could do that with a "walk" over that directory.
>
> Each plugin is supposed to be a class derived from a general
> "Plugin" superclass. I just don't know how to 'register' every
> plugin. The main application needs to know which plugin classes
> there are. On IRC I was recommended walking through all objects
> and finding out if the class is a subclass of "Plugin". Another
> person recommended using metaclasses to automatically register
> the plugin in a global list.
>
> Since I have only little real-life Python knowledge I wonder what the
> best practice for this kind of problem is.
>
> I looked at the "supybot" IRC bot to get an idea how plugins are handled
> there. Unfortunately it was still a bit over my (python) head.[/color]

I recently came up against this exact problem. My preference is to have
the plugin writer call a method to register the plugins, as this allows
him the most control. Something along these lines:

class A_plugin(Plugin ):
...

class Another_plugin( Plugin):
...

register( A_plugin )
register( Another_plugin, optional_custom _registration_p arameters )


I also like the Mark Pilgrim approach of having the plugins follow a
strict naming convention, then searching for them with hasattr or by
grepping each plugin module's namespace. E.g., I have a build script
written in python that I use instead of the "make" utility; for each
target, I define a method called make_<target> in my plugin module, like
this:

function = "make_%s" % target
if hasattr(module, function):
getattr(module, function)()
else:
print >>sys.stderr, 'Unknown target "%s"' % target

[beza1e1]

Re: Best practices for dynamically loading plugins at startup

I wrote this one:
--------------------------------------
def load_plugin(sel f, plugin, paths):
import imp
# check if we haven't loaded it already
try:
return sys.modules[plugin]
except KeyError:
pass
# ok, the load it
fp, filename, desc = imp.find_module (plugin, paths)
try:
mod = imp.load_module (plugin, fp, filename, desc)
finally:
if fp:
fp.close()
# instantiate and put into basket
clazz = mod.main(self.c onfig)
if "input" in clazz.types:
self.inputs.app end(clazz)
if "parser" in clazz.types:
self.parser.app end(clazz)
if "output" in clazz.types:
self.outputs.ap pend(clazz)
--------------------------------------
The imp module is the key:

http://docs.python.org/lib/module-imp.html

The parameters for the load module function, are found by look through
the directory. So my plugins had a convention:
They have a class called main, which is initialized with one argument,
the config object.

That is quite a simple plugin system. You can check out the whole thing
here:

https://developer.berlios.de/projects/feedisto/

[Christoph Haas]

Re: Best practices for dynamically loading plugins at startup

On Sun, Sep 25, 2005 at 11:33:03PM -0400, Jeff Schwab wrote:[color=blue]
> I recently came up against this exact problem. My preference is to have
> the plugin writer call a method to register the plugins, as this allows
> him the most control. Something along these lines:
>
> class A_plugin(Plugin ):
> ...
>
> class Another_plugin( Plugin):
> ...
>
> register( A_plugin )
> register( Another_plugin, optional_custom _registration_p arameters )[/color]

I like the idea. And "supybot" seems to do just that. What would the
"def register:" do? Maintain a global list of registered plugins?
I didn't like the idea of having global variables. Or would register
be a method of some "main" class? How would I access that?

Thanks to everybody else who posted ideas on my problem. I'm trying
all the proposals to get an idea of which approach works best.

Regards
Christoph
--
~
~
~
".signature " [Modified] 3 lines --100%-- 3,41 All

[Jeff Schwab]

Re: Best practices for dynamically loading plugins at startup

Christoph Haas wrote:[color=blue]
> On Sun, Sep 25, 2005 at 11:33:03PM -0400, Jeff Schwab wrote:
>[color=green]
>>I recently came up against this exact problem. My preference is to have
>>the plugin writer call a method to register the plugins, as this allows
>>him the most control. Something along these lines:
>>
>> class A_plugin(Plugin ):
>> ...
>>
>> class Another_plugin( Plugin):
>> ...
>>
>> register( A_plugin )
>> register( Another_plugin, optional_custom _registration_p arameters )[/color]
>
>
> I like the idea. And "supybot" seems to do just that. What would the
> "def register:" do? Maintain a global list of registered plugins?
> I didn't like the idea of having global variables. Or would register
> be a method of some "main" class? How would I access that?[/color]

The registry clearly has to be shared between modules, so I think it's
best to make it a module-level variable. In the simple-minded code
below, I've stored it in the "plugins" module. A better choice might be
the module that defines your plugins' common base class, if they have one.


# BEGIN framework.py
if __name__ == "__main__":
import plugins
print plugins.plugin_ registry
# END framework.py

# BEGIN plugins.py
# Define some plugins.

class A_plugin(object ):
def __init__(self):
print "A_plugin() "

class Another_plugin( object):
def __init__(self):
print "Another_plugin ()"

# Register the plugins.

import registry

plugin_registry = registry.Regist ry()
plugin_registry .register(A_plu gin)
plugin_registry .register(Anoth er_plugin)
# END plugins.py

# BEGIN registry.py
class Registry(object ):
"Each instance may be used to store a set of objects."

def __init__(self):
"Initialize an empty registry."
self.registered _objects = set()

def register(self, o):
"Add an object to the registry."
self.registered _objects.add(o)

def __repr__(self):
return "Registry Contents:\n\t" + \
"\n\t".join(rep r(o) for o in self.registered _objects)
# END registry.py