Web server with Python

[Patrick Useldinger]

Hi all,
after my unsuccessful try to run Apache 2 with mod_python and Python
2.3, I am looking for an alternative approach.
My aim is to write a small web-based application:

Python
- the solution must be based / usable on Python 2.3

Portability
Targeted plaforms are
- Windows 2000 Server
- Windows XP Pro
- AIX 5.1

Webserver:
- supporting up to 50 concurrent client connections
- must be able to use Python 2.3 to create dynamic pages
I can think of 3 alternatives to Apache:
- Medusa
- Twisted
- Zope
Which one is the easiest to work with? Which one is known to work on an
AIX 5.1 platform, for instance?

Persistent storage
- Is ZODB easily usable from outside Python, i.e. to produce reports
without reverting to programming?
- Is MySQL with Python a good combination?


Regards,
-PU

[Thomas Weholt]

Re: Web server with Python


"Patrick Useldinger" <p.useldinger@m yrealbox.com> wrote in message
news:tnr1mvgnt4 5urvu6n4q6dso3e to73hfr1m@4ax.c om...[color=blue]
> Hi all,
> after my unsuccessful try to run Apache 2 with mod_python and Python
> 2.3, I am looking for an alternative approach.
> My aim is to write a small web-based application:
>
> Python
> - the solution must be based / usable on Python 2.3
>
> Portability
> Targeted plaforms are
> - Windows 2000 Server
> - Windows XP Pro
> - AIX 5.1
>
> Webserver:
> - supporting up to 50 concurrent client connections
> - must be able to use Python 2.3 to create dynamic pages
> I can think of 3 alternatives to Apache:
> - Medusa
> - Twisted
> - Zope
> Which one is the easiest to work with? Which one is known to work on an
> AIX 5.1 platform, for instance?[/color]

I've worked with Zope, Medusa and Twisted. If you can use anything found on
www.zope.org under resource and like to do everything in a webpage, not
doing much programming yourself, then Zope might be the thing. It covers the
python-code a bit much for my taste ( a bit being a huge understatement ).

Medusa is easy to work with but I had some problems with blocking and
form-handling ( mainly posted files etc. ) but it might have been my lack of
understanding back then.

Twisted is very easy to get a grip on if you stay away from Woven and use
the resource-class yourself. I can produce a very simple example if you
like. I find Twisted to have a very steep learning curve in some areas and
the project do have a feeling being "under construction", but the
developers/mailing-list are very friendly and helpful and glad for input.

I haven't done anything on AIX so I don't know anything about portability.
I'd go for Twisted if you want control, Zope if you can use any existing
"product" available at zope.
[color=blue]
>
> Persistent storage
> - Is ZODB easily usable from outside Python, i.e. to produce reports
> without reverting to programming?
> - Is MySQL with Python a good combination?[/color]

MySQL is easy to work with using Python, so yes. My experience with ZODB has
not been good. I had problems compiling earlier version of it ( probably my
fault but still ) and I quickly found out that I needed SQL to be able to
store and retrieve data the way I wanted. If MySQL is too much for you take
a look at SQLite, it has a python-module and easy to install/distribute and
it is a "real" sql-database, ie. it can understand SQL and feels like a
SQL-database.

Thomas W

[Klaus Alexander Seistrup]

Re: Web server with Python

Thomas Weholt wrote:
[color=blue][color=green]
>> - must be able to use Python 2.3 to create dynamic pages[/color]
>
> If you can use anything found on www.zope.org under resource and
> like to do everything in a webpage, not doing much programming
> yourself, then Zope might be the thing.[/color]

However, Zope still doesn't run on Python 2.3, does it?


// Klaus

--[color=blue]
><> unselfish actions pay back better[/color]

[Dave Kuhlman]

Re: Web server with Python

Thomas Weholt wrote:

[snip]
[color=blue]
>
> Twisted is very easy to get a grip on if you stay away from Woven
> and use the resource-class yourself. I can produce a very simple
> example if you like. I find Twisted to have a very steep learning
> curve in some areas and the project do have a feeling being "under
> construction", but the developers/mailing-list are very friendly
> and helpful and glad for input.[/color]

The Twisted mailing list has been down for a few days. They're
moving.

To get Twisted, check:

http://www.twistedmatrix.com/

I agree with the positive comments about Twisted.

The Twisted documentation seems pretty good and fairly extensive.
And, for Web applications, here is a document I've written that
might help you to get started:

http://www.rexx.com/~dkuhlman/twisted_patterns.html

I've checked this document carefully myself, but have not yet been
able to get any Twisted experts to review it yet (again, the email
list is down temporarily), so "reader beware".

Twisted is more than Web apps, by the way.

[snip]
[color=blue]
> MySQL is easy to work with using Python, so yes. My experience
> with ZODB has not been good. I had problems compiling earlier
> version of it ( probably my fault but still ) and I quickly found
> out that I needed SQL to be able to store and retrieve data the
> way I wanted. If MySQL is too much for you take a look at SQLite,
> it has a python-module and easy to install/distribute and it is a
> "real" sql-database, ie. it can understand SQL and feels like a
> SQL-database.[/color]

PostgreSQL also seems to work well with Twisted. See my document
(mentioned above) for code examples on using PostgreSQL (or other
relational databases, with minor changes) with Twisted.

Dave

--
Dave Kuhlman

http://www.rexx.com/~dkuhlman

dkuhlman@rexx.c om

[Geoff Gerrietts]

Re: Web server with Python

Quoting Patrick Useldinger (p.useldinger@m yrealbox.com):[color=blue]
> Webserver:
> - Medusa
> - Twisted
> - Zope
> Which one is the easiest to work with? Which one is known to work on an
> AIX 5.1 platform, for instance?[/color]

Medusa is pretty low-level. It's really just a server framework with
not much in the way of form handling or other niceties.

Zope is pretty high-level. It's at its best in doing content
management, and does a decent job of being an appserver. Its biggest
shortcomings, in my disreputable opinion, are its use of an object
database and its innovative features.

The object database makes it hard to edit stuff except in the web
interface. Several of the innovative features -- security roles, user
authorization, and acquisition to name a few -- make serious,
highly-dynamic development under Zope challenging. You are effectively
learning the intricacies of a highly-evolved application. On the other
hand, if your applications fit into typical categories -- content
management, information publishing, forums, etc -- Zope has lots of
pre-built tools that you can use.

Also, it really depends on how dynamic your "dynamic" is. A friend of
mine runs a job database: not too many writes to the DB and lots of
reads from it, a couple common data types. Zope would be okay for his
stuff. I work on a site where every user submits a profile, and lots
of information is exchanged between users, and every page's content is
entirely different based on the current user's nest of objects. We had
to punch a hole in the side of Zope and we suffer from performance
problems due to the highly dynamic content.

I've looked at Twisted but not used it. It's got some of the low-level
features that Medusa does, but it's also got some of the higher-level
form handling and such that you get out of Zope. It's worth a look.

Also worth a look is WebWare. I've only done a little bit with
WebWare, but it's very similar in architecture and execution to the
Java Servlet API. If I were starting from scratch, that would be my
toolkit of choice for a highly stateful, highly dynamic app.

I can't speak to AIX. You might be the first to try some of these on
those platforms. My advice in those cases is to look at the stuff that
uses as little C code as possible, because gauging portability isn't
too easy.
[color=blue]
> Persistent storage
> - Is ZODB easily usable from outside Python, i.e. to produce reports
> without reverting to programming?[/color]

No. ZODB is an object database. You put objects in and build your own
indexes to those objects. You get them out the same way you put them
in. ZODB is easily usable from /inside/ Python though: it does not
require you to explicitly handle the object-to-relational mapping that
a SQL database will.
[color=blue]
> - Is MySQL with Python a good combination?[/color]

I can't comment on MySQL directly, because I've used it only a little
from Python. SQL in general suffers from the difficulty that you need
to write the queries to store and retrieve your objects from the
databse yourself: that process is called object-to-relational mapping,
and it's hard to do well and easy to do in ways you'll regret later.
But, the SQL databases are easily accessible from a quadrillion other
tools that can do reporting for you.

Hope this helps,
--G.

--
Geoff Gerrietts "I think doorways are sexist."
<geoff at gerrietts net> --Jayson Lockwood

[Geoff Gerrietts]

Re: Web server with Python

Quoting Klaus Alexander Seistrup (spam@magnetic-ink.dk):[color=blue]
> Thomas Weholt wrote:
>[color=green][color=darkred]
> >> - must be able to use Python 2.3 to create dynamic pages[/color]
> >
> > If you can use anything found on www.zope.org under resource and
> > like to do everything in a webpage, not doing much programming
> > yourself, then Zope might be the thing.[/color]
>
> However, Zope still doesn't run on Python 2.3, does it?[/color]

Also true! Beta versions recommend/require 2.2.3, and current
production last time I checked (last week? week before?) requires
2.1.3 instead.

--G.

--
Geoff Gerrietts "Overspecia lize and you breed in weakness.
<geoff at gerrietts net> It's slow death." -- Maj. Motoko Kusanagi
http://www.gerrietts.net (Ghost in the Shell)

[Andrew Bennetts]

Re: Web server with Python

On Thu, Sep 11, 2003 at 05:19:50PM -0700, Dave Kuhlman wrote:[color=blue]
>
> The Twisted documentation seems pretty good and fairly extensive.
> And, for Web applications, here is a document I've written that
> might help you to get started:
>
> http://www.rexx.com/~dkuhlman/twisted_patterns.html
>
> I've checked this document carefully myself, but have not yet been
> able to get any Twisted experts to review it yet (again, the email
> list is down temporarily), so "reader beware".[/color]

Here's some quick feedback for you in the meantime, then :)

] Here is an example of a class that dispatches requests:
]
] class ResourceDispatc her(resource.Re source):
]
] isLeaf = True
]
] def render(self, request):
] done = False
] if len(request.pos tpath) == 1 and request.postpat h[0]:
] if request.postpat h[0] == 'show_plants':
] dbaccess = dbmod.DBAccess( registry)
] dbaccess.show_p lants(request)
] done = True
] elif request.postpat h[0] == 'show_plants_by _rating':
] dbaccess = dbmod.DBAccess( registry)
] dbaccess.show_p lants_by_rating (request)
] done = True
[...etc...]

Ugh. You can do this with much less pain:

class ResourceDispatc her(resource.Re source):
isLeaf = True
def render(self, request):
if len(request.pos tpath) == 1 and request.postpat h[0]:
name = request.postpat h[0]
dbaccess = dbmod.DBAccess( registry)
try:
method = getattr(dbacces s, 'webrequest_' + name)
except AttributeError:
pass
else:
method(request)
return NOT_DONE_YET
content = Content_Dispatc h % (
request.postpat h,
request.args,
time.ctime(),
)
request.write(c ontent)
request.finish( )

Note that the getattr-with-a-prefix idiom is used in several places in
Twisted, e.g. twisted.web.xml rpc and twisted.spread. pb, among others.

] def delete_plant_st ep_3(self, resultlist):
] self.db.getPlan ts().addCallbac ks(self.gotPlan ts,
] self.db.operati onError,
] callbackArgs=() )

callbackArgs defaults to (), there's no need to explicitly pass it.

] def add_plant_machi ne(self, *args):
] if self.state == STEP_1:
] self.db = PlantDatabase(s elf.dbpool)

Again, prefixed methods make this much nicer:

class Foo:
state = 'step1'
def add_plant_machi ne(self, *args):
method = getattr(self, 'state_' + self.state, self.invalidSta te)(*args)

def state_step1(sel f, *args):
self.db = PlantDatabase(s elf.dbpool)
# ...etc...
self.state = 'step2'

def state_step2(sel f, *args):
# And so on...

This is much nicer way to construct a state machine than a massive
if/elif/elif/elif/... block. A good example of this in Twisted is
twisted.protoco ls.smtp (and there's a helper class at
twisted.protoco ls.basic.Statef ulStringProtoco l -- but that's not really
relevant to web programming)

Even nicer would be thinking of meaningful names for the states instead of
just "step1", etc :)

] class PlantDatabase(a dbapi.Augmentat ion):

Augmentation is deprecated, and will die soon. You're better off just using
adbapi.Connecti onPool directly.

] class GlimpseTextRepo sitory:
] """Update and retrieve from the Plant_DB database.
] """
] def __init__(self):
] pass

Why not just omit the redundant __init__ definition entirely?

] # ..snip..
] self.deferred.a ddCallback(self .got_query_resu lts)
] # ..snip..
]
] def got_query_resul ts(self, results):
] return results

Your comment below the code is right; you can just omit this entirely. That
callback is redundant.

] os.system(cmd)

This is a no-no in Twisted -- it will block. Use the methods like
twisted.interne t.utils.getProc essOutput instead.

] #
] # XML-RPC applicationLogi c class
] #
] class TemperatureAcce ss:
[..snip..]
] #
] # Produce a form to be used to search the text repository.
] #
] def show_temperatur e_form(self, request):
] dbglogmsg('*** (show_search_fo rm)')
] self.request = request
] self.content = Content_Convert TemperatureForm
] self.request.wr ite(self.conten t)
] self.request.fi nish()
] return NOT_DONE_YET

This is a pretty messy seperation of application logic from the XML-RPC
layer -- you're operating on a HTTPRequest object, which has nothing to do
with temperature logic. Your abstractions are leaking.

Your example here is pretty confusing. I *think* what you're doing is
having a Twisted webserver render requests based on data retrieved
dynamically via XML-RPC, but I can't see anywhere where you state that
clearly. It took me a while to figure out that TemperatureAcce ss was on the
XML-RPC *client* side, not the server.

I recommend trying to rewrite this example using Woven. It provides a
framework that encourages proper seperation of Models (the data structures,
whereever they come from, in this case XML-RPC) and Views (the way the model
is presented), and think it would come out much more elegantly.

I also recommend rewriting your example XML-RPC server using
twisted.web.xml rpc.XMLRPC -- it provides a very easy way to write XML-RPC
servers, and seeing as this is a document about Twisted, it may as well use
it :) You'll probably find the example in doc/examples/xmlrpc.py to be
helpful.

Finally, for the test harness, you might want to consider re-using the
infrastructure in Twisted, rather than rolling your own. For a brief
example, I recommend the DeferredModelTe stCase in
twisted/test/test_woven.py.

I hope this feedback helps.

-Andrew.

[Patrick Useldinger]

Re: Web server with Python

On Thu, 11 Sep 2003 17:19:50 -0700, Dave Kuhlman <dkuhlman@rexx. com>
wrote:
[color=blue]
>
>Twisted is more than Web apps, by the way.[/color]

Does Twisted work well with Python 2.3?

-pu

[Andrew Bennetts]

Re: Web server with Python

On Fri, Sep 12, 2003 at 09:21:58AM +0200, Patrick Useldinger wrote:[color=blue]
> On Thu, 11 Sep 2003 17:19:50 -0700, Dave Kuhlman <dkuhlman@rexx. com>
> wrote:
>[color=green]
> >
> >Twisted is more than Web apps, by the way.[/color]
>
> Does Twisted work well with Python 2.3?[/color]

Yes, current versions support 2.3 without any problems.

-Andrew.

[pd]

Re: Web server with Python

How about skunkweb? I haven't used this, in fact I'm only just getting
into python at the moment. Perhaps someone with experience can comment
on this app...
To quote the site, skunkweb ... "is an extensible, scalable and easy to
use application server written in Python"

hth

Paul.
p3d AT yahoo dot co dot uk


Patrick Useldinger wrote:[color=blue]
> Hi all,
> after my unsuccessful try to run Apache 2 with mod_python and Python
> 2.3, I am looking for an alternative approach.
> My aim is to write a small web-based application:
>
> Python
> - the solution must be based / usable on Python 2.3
>
> Portability
> Targeted plaforms are
> - Windows 2000 Server
> - Windows XP Pro
> - AIX 5.1
>
> Webserver:
> - supporting up to 50 concurrent client connections
> - must be able to use Python 2.3 to create dynamic pages
> I can think of 3 alternatives to Apache:
> - Medusa
> - Twisted
> - Zope
> Which one is the easiest to work with? Which one is known to work on an
> AIX 5.1 platform, for instance?
>
> Persistent storage
> - Is ZODB easily usable from outside Python, i.e. to produce reports
> without reverting to programming?
> - Is MySQL with Python a good combination?
>
>
> Regards,
> -PU[/color]

[Patrick Useldinger]

Re: Web server with Python

On Fri, 12 Sep 2003 17:33:34 +1000, Andrew Bennetts
<andrew-pythonlist@puzz ling.org> wrote:
[color=blue][color=green]
>> Does Twisted work well with Python 2.3?[/color]
>
>Yes, current versions support 2.3 without any problems.[/color]

Version 1.0.6 refuses to install, though, complaining it requires Python
2.2.

-PU

[Patrick Useldinger]

Re: Web server with Python

On Fri, 12 Sep 2003 12:50:13 +0100, pd <p@d.com> wrote:
[color=blue]
>How about skunkweb? I haven't used this, in fact I'm only just getting
>into python at the moment. Perhaps someone with experience can comment
>on this app...[/color]

The homepage says it doesn't work on Win32, so it's not a good candidate
for me.
-PU

[Harri Pesonen]

Re: Web server with Python

AOLserver:

http://www.aolserver.com/

PyWX: Python for AOLserver:

http://pywx.idyll.org/

"PyWX is a plug-in for the open source Web server AOLserver that embeds
a Python interpreter within the AOLserver process. This allows AOLserver
to execute Python scripts in response to HTTP requests in a variety of ways.

AOLserver is a high-performance, single-process/multiple-threads Web
server that is used at several high-throughput sites including AOL.com
and digitalcity.com . PyWX enables full access to the AOLserver API from
Python, as well as tying PyWX into AOLserver as an almost complete peer
to Tcl."

Unfortunately Python is not free-threading, so it is not ideal for web
server.

Harri

Patrick Useldinger wrote:
[color=blue]
> Hi all,
> after my unsuccessful try to run Apache 2 with mod_python and Python
> 2.3, I am looking for an alternative approach.
> My aim is to write a small web-based application:
>
> Python
> - the solution must be based / usable on Python 2.3
>
> Portability
> Targeted plaforms are
> - Windows 2000 Server
> - Windows XP Pro
> - AIX 5.1
>
> Webserver:
> - supporting up to 50 concurrent client connections
> - must be able to use Python 2.3 to create dynamic pages
> I can think of 3 alternatives to Apache:
> - Medusa
> - Twisted
> - Zope
> Which one is the easiest to work with? Which one is known to work on an
> AIX 5.1 platform, for instance?
>
> Persistent storage
> - Is ZODB easily usable from outside Python, i.e. to produce reports
> without reverting to programming?
> - Is MySQL with Python a good combination?
>
>
> Regards,
> -PU[/color]

[Patrick Useldinger]

Re: Web server with Python

On Sat, 13 Sep 2003 08:06:39 -0500, "A.M. Kuchling" <amk@amk.ca> wrote:

[color=blue]
>Quixote (www.mems-exchange.org/software/quixote, www.quixote.ca) comes with
>modules to work with both Medusa and Twisted (and also AOLserver and Apache).[/color]

I'll have a look at it, thanks.
[color=blue][color=green]
>> - supporting up to 50 concurrent client connections[/color]
>
>Do you mean 50 hits per second, which is a substantial load (4 million hits
>a day!) or just 50 users on the site at the same time, which is a hit every
>second or two? Anything should be able to handle the latter case; I'd go
>with Apache + SCGI for the former.
>[/color]

50 users at the same time, a very moderate load I believe.
BTW, after trying mod_pyton (KO due to Python 2.3), Twisted (idem.), I
am currently playing around with Medusa and I like it. I think I am
going into that direction. Thanks for keeping it alive.

-PU

[Andrew Bennetts]

Re: Web server with Python

[CC'ing the Twisted mailing list]

On Fri, Sep 12, 2003 at 08:05:59PM +0200, Patrick Useldinger wrote:[color=blue]
> On Fri, 12 Sep 2003 17:33:34 +1000, Andrew Bennetts
> <andrew-pythonlist@puzz ling.org> wrote:
>[color=green][color=darkred]
> >> Does Twisted work well with Python 2.3?[/color]
> >
> >Yes, current versions support 2.3 without any problems.[/color]
>
> Version 1.0.6 refuses to install, though, complaining it requires Python
> 2.2.[/color]

Eek! Does 1.0.7 (just released) work any better? Also, what platform are
you using, and are you installing from the source .tar.gz or something else?

-Andrew.