Pyparsing Question

[Marc 'BlackJack' Rintsch]

Re: scaling problems

On Tue, 20 May 2008 10:47:50 +1000, James A. Donald wrote:

2. It is not clear to me how a python web application scales.

Ask YouTube. :-)

Ciao,
Marc 'BlackJack' Rintsch

[Graham Dumpleton]

Re: scaling problems

On May 20, 2:00 pm, James A. Donald <jam...@echeque .comwrote:

2.  It is not clear to me how a python web application scales.  Python
is inherently single threaded, so one will need lots of python
processes on lots of computers, with the database software handling
parallel accesses to the same or related data.  One could organize it
as one python program for each url, and one python process for each
http request, but that involves a lot of overhead starting up and
shutting down python processes.  Or one could organize it as one
python program for each url, but if one gets a lot of http requests
for one url, a small number of python processes will each sequentially
handle a large number of those requests.  What I am really asking is:
Are there python web frameworks that scale with hardware and how do
they handle scaling?

>
Reid Priedhorsky
>

This sounds like a good match for Apache withmod_python.

>
I would hope that it is, but the question that I would like to know is
how does mod_python handle the problem - how do python programs and
processes relate to web pages and http requests when one is using mod_python, and what happens when one has quite a lot of web pages and
a very large number of http requests?

Read:

http://blog.dscpl.com.au/2007/09/parallel-python-discussion-and-modwsgi.html

http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading

They talk about multi process nature of Apache and how GIL is not as
big a deal when using it.

The latter document explains the various process/threading modes when
using Apache/mod_wsgi. The embedded modes described in that
documentation also apply to mod_python.

The server is generally never the bottleneck, but if you are paranoid
about performance, then also look at relative comparison of mod_wsgi
and mod_python in:

http://code.google.com/p/modwsgi/wiki/PerformanceEstimates

Graham

[Nick Craig-Wood]

Re: scaling problems

Marc 'BlackJack' Rintsch <bj_666@gmx.net wrote:

On Tue, 20 May 2008 13:57:26 +1000, James A. Donald wrote:
>

The larger the program, the greater the likelihood of inadvertent name
collisions creating rare and irreproducible interactions between
different and supposedly independent parts of the program that each
work fine on their own, and supposedly cannot possibly interact.

>
How should such collisions happen? You don't throw all your names into
the same namespace!?

If you ever did a lot of programming in C with large projects you have
exactly that problem a lot - there is only one namespace for all the
external functions and variables, and macro definitions from one
include are forever messing up those from another. I suspect the OP
is coming from that background.

However python doesn't have that problem at all due to its use of
module namespaces - each name is confined to within a module (file)
unless you take specific action otherwise, and each class attribute is
confined to the class etc.

From the Zen of Python "Namespaces are one honking great idea -- let's
do more of those!" - as a battle scarred C programmer I'd agree ;-)

--
Nick Craig-Wood <nick@craig-wood.com-- http://www.craig-wood.com/nick

[Duncan Booth]

Re: scaling problems

Marc 'BlackJack' Rintsch <bj_666@gmx.net wrote:

On Tue, 20 May 2008 10:47:50 +1000, James A. Donald wrote:
>

>2. It is not clear to me how a python web application scales.

>
Ask YouTube. :-)

Or look at Google appengine where unlike normal Python you really are
prevented from making good use of threading.

Google App Engine takes an interesting approach by forcing the programmer
to consider scalability right from the start: state is stored in a
distributed database which cannot do all the hard to scale things that SQL
databases do. This means that you have to work as though your application
were spread on servers all round the world from day 1 instead of waiting
until the structure that was 'good enough' is threatening to kill your
business before you address them.

It also puts strict limits on how much a single web request can do, so
again you have to work from day 1 to make sure that page requests are as
efficient as possible.

In return you get an application which should scale well. There is nothing
Python specific about the techniques, it is just that Python is the first
(and so far only) language supported on the platform.

--
Duncan Booth http://kupuguy.blogspot.com

[David Stanek]

Re: scaling problems

On Tue, May 20, 2008 at 12:03 AM, James A. Donald <jamesd@echeque .comwrote:

On Mon, 19 May 2008 21:04:28 -0400, "David Stanek"
<dstanek@dstane k.comwrote:

>What is the difference if you have a process with 10 threads or 10
>separate processes running in parallel? Apache is a good example of a
>server that may be configured to use multiple processes to handle
>requests. And from what I hear is scales just fine.
>>
>I think you are looking at the problem wrong. The fundamentals are the
>same between threads and processes.

>
I am not planning to write a web server framework, but to use one.
Doubtless a python framework could be written to have satisfactory
scaling properties, but what are the scaling properties of the ones
that have been written?
>

Both Django and TurborGears work well for me. When you step back and
think about it all of the popular web frameworks would do just fine.
The ones that don't do multiprocessing out of the box would be trivial
to load balance behind Apache or a real load balancer. Again the
problem here is the number of connections to the database, once you
get big enough to worry about it.

--
David

http://www.traceback.org