Python does not play well with others

[Paul Boddie]

Re: Python does not play well with others

John Nagle wrote:

>
The real problems are with integration of modules that
aren't written in Python. Python-only modules don't have the
version compatibility problems which C modules do. Loading
a Python-only module from an external source is usually not
a big deal. Building a C module, especially one with
dependencies on other components, can be a big deal.
So, focus on problems with modules which have C
components.

I think that the integration of certain "popular" extensions into the
Python distribution has been one of the tools employed to mitigate the
version compatibility situation: if the popular stuff gets released
with Python, perhaps fewer people will complain about "C modules"
whose developers/maintainers haven't updated them for the latest
release. I'm not really convinced that this makes for a sustainable
approach, however, since this just becomes a process of accretion
where the core development community must get larger and become very
well-coordinated in order to manage the dependencies within the
software, as well as tracking external dependencies.

Over the years there have been suggestions which might have made the
compatibility situation a bit better: a "Python in a tie" release was
an objective of the seemingly dormant Python Business Foundation;
Linux Standard Base inclusion of Python has been proposed and
discussed. I can't help feeling that without improvements in such
other areas, things like "C module" availability for different Python
versions (and for different versions of related libraries) is mainly
an issue of doing the almost thankless hard work of backporting,
testing and managing different build configurations. Still, there may
be demand for a backports community in all this.

Paul

[Graham Dumpleton]

Re: Python does not play well with others

On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:

"Paul Boddie" <p...@boddie.or g.ukwrites:

>Probably the biggest inhibitor, as far as I can see, has been the
>server technology chosen. Many hosting providers have historically
>offered no better than CGI for Python, whilst PHP runs within Apache
>itself, and it has previously been stated that mod_python has been
>undesirable with regard to isolating processes from each other.
>Consequently , a number of Python people seem to have held out for
>other "high performance" solutions, which various companies now offer.

>
Your point that shared hosting with Python isn't so easy because of
insufficient isolation between apps is valid. Maybe Python 3.0 can do
something about that and it seems like a valid thing to consider while
fleshing out the 3.0 design.

To clarify some points about mod_python, since these posts do not
properly explain the reality of the situation and I feel people are
getting the wrong impression.

First off, when using mod_python it is possible to have it create
multiple sub interpreters within each Apache child process. These
distinct sub interpreters can be linked to different parts of the URL
namespace. This means that it is possible to host more than one
mod_python application where each executes within in their own
distinct sub interpreter. The outcome of this is that each application
can have their own sys.path, own os.environ, own sets of modules and
potentially with different versions of some module.

Maintaining separation using sub interpreters eliminates the bulk of
problems with applications interfering which each other at least
within a process. Some problems can still arise though where third
party extension modules for Python aren't written so as to be usable
from multiple sub interpreters at the same time however. This is not a
failing of mod_python though, but a failing of the module writers.

The main area where interference can occur is where applications needs
to write to the file system. This is because all code will be
executing as the user that Apache runs as. Thus, distinct applications
could overwrite each others data within the file system. On one level
this just means that applications need to be configured to always use
their own part of the file system. For example, if using the support
in mod_python for sessions, the distinct applications should perhaps
use separate session databases rather than use the same common
database. Ultimately though, a rogue application could write where
ever it wants to, but from what I know (and could be wrong), this
isn't different to other languages systems within Apache such as PHP
and mod_perl.

Another possibility for interference is where an application simply
does something bad like get stuck in a tight loop or consume lots of
memory. Such an event can possibly interfere with other applications,
even across language boundaries, however, how bad the impact will be
depend on what MPM is used by Apache.

If "prefork" MPM is used, then the request being handled by that
application is the only thing which would be running within that child
process at that particular time. Thus, if it stops the functioning of
just that one process it doesn't matter as Apache will just farm
requests off to other child processes. If that initial child process
crashes because of the problem, then again it doesn't matter as Apache
will just create another child process to replace it and will
otherwise keep running.

If the "worker" MPM is used the impact can be greater as there could
be other requests being handled concurrently within the same child
process and the performance of those requests may be hindered. If the
worst happens and the child process crashes, only other requests being
handled within that specific child process would be affected, those in
other child processes would again continue unaffected as would Apache
as a whole.

The worst case is the "winnt" MPM (ie., Windows boxes). This is
because there is only one Apache process and thus a rogue application
can affect the whole Apache web server.

It should be highlighted though that this class of problem is not
unique to Python or mod_python as you could get rogue code in a PHP
page or mod_perl application just as easily.

What it all really comes down to is that the only features that are
really missing are the ability for distinct applications to run as
distinct users, or for applications to run inside of some sort of
chroot environment. Some aspects of this are addressed by FCGI and
SCGI, but again lack of this feature within mod_python itself is not
unique to it and that as far as I know is also going to be an issue
for other language systems for Apache such as PHP or mod_perl.

Having said all that, perhaps those who are complaining about lack of
support for specific features in mod_python can now clarify what
actually you are talking about. At the moment the brief comments being
made seem to possibly cover some things that mod_python can already do
but may not be obvious.

Graham

[Paul Boddie]

Re: Python does not play well with others

Graham Dumpleton wrote:

>
Having said all that, perhaps those who are complaining about lack of
support for specific features in mod_python can now clarify what
actually you are talking about. At the moment the brief comments being
made seem to possibly cover some things that mod_python can already do
but may not be obvious.

I had a more complete response to this before Google Groups and my
browser went into "stupid mode" together, but I'd like to say that I
don't have any complaints about mod_python, but I know that there were
people who claimed [1] that PHP had its "safe mode" [2] which gave
isolation to applications belonging to different users in the same
Apache instance, asserting that mod_python not having that particular
feature prevented wider adoption of Python in the hosting business.

It's good to see a clarification of mod_python and its position in
relation to such issues, though. I don't really share the concerns
mentioned, and wouldn't want to be using or providing services of the
kind previously discussed, anyway. Instead, I'd want to deploy
applications using other techniques such as virtualisation, rather
than throwing customers into the same Web server and, apparently in
the case of PHP, relying on hearsay about whether they can interfere
with each other. I note that the creator of mod_python seems to have a
similar perspective on virtualisation, given the nature of his hosting
company's services.

Paul

[1] http://groups.google.com/group/comp.lang.python/msg/
469dd47f5c1ad52 1
[2] http://no.php.net/features.safe-mode

[John Nagle]

Re: Python does not play well with others

Graham Dumpleton wrote:

On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:
>

>>"Paul Boddie" <p...@boddie.or g.ukwrites:
>>

>>>Probably the biggest inhibitor, as far as I can see, has been the
>>>server technology chosen. Many hosting providers have historically
>>>offered no better than CGI for Python, whilst PHP runs within Apache
>>>itself, and it has previously been stated that mod_python has been
>>>undesirabl e with regard to isolating processes from each other.
>>>Consequently , a number of Python people seem to have held out for
>>>other "high performance" solutions, which various companies now offer.

>>
>>Your point that shared hosting with Python isn't so easy because of
>>insufficien t isolation between apps is valid. Maybe Python 3.0 can do
>>something about that and it seems like a valid thing to consider while
>>fleshing out the 3.0 design.

>
>
To clarify some points about mod_python, since these posts do not
properly explain the reality of the situation and I feel people are
getting the wrong impression.
>
First off, when using mod_python it is possible to have it create
multiple sub interpreters within each Apache child process.

Realistically, mod_python is a dead end for large servers,
because Python isn't really multi-threaded. The Global Python
Lock means that a multi-core CPU won't help performance.

FastCGI, though, can get all the CPUs going. It takes more
memory, though, since each instance has a full copy of Python
and all the libraries in use.

(FastCGI is a straightforward transaction processing engine.
Each transaction program is launched in a separate process, and,
once done with one transaction, can be used to do another one
without reloading. When things are slow, the extra transaction processes
are told to exit; when load picks up, more of them are forked.
Security is comparable to CGI.)

John Nagle

[Graham Dumpleton]

Re: Python does not play well with others

On Feb 5, 9:45 am, John Nagle <n...@animats.c omwrote:

Graham Dumpleton wrote:

On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:

>

>"Paul Boddie" <p...@boddie.or g.ukwrites:

>

>>Probably the biggest inhibitor, as far as I can see, has been the
>>server technology chosen. Many hosting providers have historically
>>offered no better than CGI for Python, whilst PHP runs within Apache
>>itself, and it has previously been stated that mod_python has been
>>undesirable with regard to isolating processes from each other.
>>Consequentl y, a number of Python people seem to have held out for
>>other "high performance" solutions, which various companies now offer.

>

>Your point that shared hosting with Python isn't so easy because of
>insufficient isolation between apps is valid. Maybe Python 3.0 can do
>something about that and it seems like a valid thing to consider while
>fleshing out the 3.0 design.

>

To clarify some points about mod_python, since these posts do not
properly explain the reality of the situation and I feel people are
getting the wrong impression.

>

First off, when using mod_python it is possible to have it create
multiple sub interpreters within each Apache child process.

>
Realistically, mod_python is a dead end for large servers,
because Python isn't really multi-threaded. The Global Python
Lock means that a multi-core CPU won't help performance.

That is not true if 'prefork' MPM is used for Apache which is how most
people seem to run it. This is because each Apache child process only
run one request at a time and so there isn't normally going to be any
contention on the GIL at all. The only case where there would be
contention in this arrangement is if the request handlers within
Apache had spawned off distinct threads themselves to do stuff. Even
then, in this arrangement the main request handler is usually not
doing anything and is just waiting for the created thread to finish
what it was doing. Thus if only one thread was spawned to do some work
or a blocking operation to allow the main thread to timeout, then
again there isn't really any contention as only one thread is actually
doing anything. If you are trying to embed very intensive operations
with threads within Apache then I would suggest it is not really the
best design you could use anyway as such things would be much better
farmed off to a long running backend process using XML-RPC or some
other interprocess communication mechanism.

If one is using the "worker" MPM then yes there will be some
contention if multiple requests are being handled by mod_python at the
same time within the same Apache child process. The downside of this
is lessened however by the fact that there are still multiple Apache
child processes and Apache will spread requests across all the Apache
child processes, thus the amount that may be running concurrently
within any one process is less.

The basic problem of GIL contention here is no different to a single
Python backend process which is handling everything behind Apache. In
some respects the Apache approach actually works better as there are
multiple processes spreading the load. Your comment on the GIL is
therefore partly unjustified in that respect for Apache and
mod_python. Your statement in some respect still stands for Python
itself when run as a single process, but you linked it to mod_python
and Apache which lessens the impact through its architecture of using
multiple child processes.

Finally we have 'winnt' MPM, again, because this is all in the one
process you will have GIL contention in a much more substantial
manner. However, I'd suggest that most wouldn't choose Apache on
Windows as a major deployment platform.

FastCGI, though, can get all the CPUs going. It takes more
memory, though, since each instance has a full copy of Python
and all the libraries in use.

How is that any different to Apache child processes. Each Apache child
process has a full copy of Python and the libraries in use. Each
Apache child process can be making use of different CPUs. Further,
static file requests, plus other requests against PHP, mod_perl etc
can when mod_python is also running be on separate CPUs within the
same child process when 'worker' MPM is being used. Thus you haven't
lost all forms or parallelism that may be possible, it is only within
the mod_python world that there will be some GIL contention and only
with 'worker' and 'winnt' MPMs, not 'prefork'. It isn't going to lock
out non Python stuff from making use of additional CPUs.

(FastCGI is a straightforward transaction processing engine.
Each transaction program is launched in a separate process, and,
once done with one transaction, can be used to do another one
without reloading. When things are slow, the extra transaction processes
are told to exit; when load picks up, more of them are forked.
Security is comparable to CGI.)

Apache will also kill off excess child processes when it deems they
are no longer required, or create new ones as demand dictates.

So, I am still not sure where the big issue is, the architecture of
Apache limits the impact of GIL contention in ways that Python alone
doesn't.

Graham

[sjdevnull@yahoo.com]

Re: Python does not play well with others

John Nagle wrote:

Graham Dumpleton wrote:

On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:

>"Paul Boddie" <p...@boddie.or g.ukwrites:
>
>>Probably the biggest inhibitor, as far as I can see, has been the
>>server technology chosen. Many hosting providers have historically
>>offered no better than CGI for Python, whilst PHP runs within Apache
>>itself, and it has previously been stated that mod_python has been
>>undesirable with regard to isolating processes from each other.
>>Consequentl y, a number of Python people seem to have held out for
>>other "high performance" solutions, which various companies now offer.
>
>Your point that shared hosting with Python isn't so easy because of
>insufficient isolation between apps is valid. Maybe Python 3.0 can do
>something about that and it seems like a valid thing to consider while
>fleshing out the 3.0 design.

To clarify some points about mod_python, since these posts do not
properly explain the reality of the situation and I feel people are
getting the wrong impression.

First off, when using mod_python it is possible to have it create
multiple sub interpreters within each Apache child process.

>
Realistically, mod_python is a dead end for large servers,
because Python isn't really multi-threaded. The Global Python
Lock means that a multi-core CPU won't help performance.

The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting.

[John Nagle]

Re: Python does not play well with others

sjdevnull@yahoo .com wrote:

John Nagle wrote:
>

>>Graham Dumpleton wrote:
>>

>>>On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:
>>>
>>>
>>>>"Paul Boddie" <p...@boddie.or g.ukwrites:

> Realistically, mod_python is a dead end for large servers,
>>because Python isn't really multi-threaded. The Global Python
>>Lock means that a multi-core CPU won't help performance.

>
>
The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting.

Pre-forking doesn't reduce load; it just improves responsiveness.
You still pay for loading all the modules on every request. For
many AJAX apps, the loading cost tends to dominate the transaction.

FastCGI, though, reuses the loaded Python (or whatever) image.
The code has to be written to be able to process transactions
in sequence (i.e. don't rely on variables intitialized at load),
but each process lives for more than one transaction cycle.
However, each process has a copy of the whole Python system,
although maybe some code gets shared.

John Nagle

[Paul Boddie]

Re: Python does not play well with others

On 5 Feb, 18:52, John Nagle <n...@animats.c omwrote:

>
Pre-forking doesn't reduce load; it just improves responsiveness.
You still pay for loading all the modules on every request. For
many AJAX apps, the loading cost tends to dominate the transaction.

According to the Apache prefork documentation, you can configure how
often the child processes stick around...

http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild

....and I suppose mod_python retains its state in a child process as
long as that process is running:

"Once created, a subinterpreter will be reused for subsequent
requests. It is never destroyed and exists until the Apache process
dies."
- http://www.modpython.org/live/curren...i-interps.html

"Depending on the use of various PythonInter* directives, a single
python interpreter (and list of imported modules, and per-module
global variables, etc) might be shared between multiple mod_python
applications."
- http://www.modpython.org/FAQ/faqw.py...=faq03.005.htp

The FAQ entry (3.1) about module reloading would appear to be
pertinent here, too:

http://www.modpython.org/FAQ/faqw.py?req=show&file=faq03.001.htp

Paul

[Paul Rubin]

Re: Python does not play well with others

John Nagle <nagle@animats. comwrites:

The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting.

>
Pre-forking doesn't reduce load; it just improves responsiveness.
You still pay for loading all the modules on every request. For
many AJAX apps, the loading cost tends to dominate the transaction.

I think the idea is that each pre-forked subprocess has its own
mod_python that services multiple requests serially.

New to me is the idea that you can have multiple separate Python
interpreters in a SINGLE process (mentioned in another post). I'd
thought that being limited to one interpreter per process was a
significant and hard-to-fix limitation of the current CPython
implementation that's unlikely to be fixed earlier than 3.0.

[Graham Dumpleton]

Re: Python does not play well with others

On Feb 6, 4:52 am, John Nagle <n...@animats.c omwrote:

sjdevn...@yahoo .com wrote:

John Nagle wrote:

>

>Graham Dumpleton wrote:

>

>>On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:

>

>>>"Paul Boddie" <p...@boddie.or g.ukwrites:
Realistically, mod_python is a dead end for large servers,
>because Python isn't really multi-threaded. The Global Python
>Lock means that a multi-core CPU won't help performance.

>

The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting.

>
Pre-forking doesn't reduce load; it just improves responsiveness.
You still pay for loading all the modules on every request. For
many AJAX apps, the loading cost tends to dominate the transaction.
>
FastCGI, though, reuses the loaded Python (or whatever) image.
The code has to be written to be able to process transactions
in sequence (i.e. don't rely on variables intitialized at load),
but each process lives for more than one transaction cycle.
However, each process has a copy of the whole Python system,
although maybe some code gets shared.

As someone else pointed out, your understanding of how mod_python
works within Apache is somewhat wrong. I'll explain some things a bit
further to make it clearer for you.

When the main Apache process (parent) is started it will load all the
various Apache modules including that for mod_python. Each of these
modules has the opportunity to hook into various configuration phases
to perform actions. In the case of mod_python it will hook into the
post config phase and initialise Python which will in turn setup all
the builtin Python modules.

When Apache forks off child processes each of those child processes
will inherit Python already in an initialised state and also the
initial Python interpreter instance which was created, This therefore
avoids the need to perform initialisation of Python every time that a
child process is created.

In general this initial Python interpreter instance isn't actually
used though, as the default strategy of mod_python is to allocate
distinct Python interpreter instances for each VirtualHost, thereby at
least keeping applications running in distinct VirtualHost containers
to be separate so they don't interfere with each other.

Yes, these per VirtualHost interpreter instances will only be created
on demand in the child process when a request arrives which
necessitates it be created and so there is some first time setup for
that specific interpreter instance at that point, but the main Python
initialisation has already occurred so this is minor. Most
importantly, once that interpreter instance is created for the
specific VirtualHost in the child process it is retained in memory and
used from one request to the next. If the handler for a request loads
in Python modules, those Python modules are retained in memory and do
not have to be reloaded on each request as you believe.

If you are concerned about the fact that you don't specifically know
when an interpreter instance will be first created in the child
process, ie., because it would only be created upon the first request
arriving that actually required it, you can force interpreter
instances to be created as soon as the child process has been created
by using the PythonImport directive. What this directive allows you to
do is specify a Python module that should be preloaded into a specific
interpreter instance as soon as the child process is created. Because
the interpreter will need to exist, it will first be created before
the module is loaded thereby achieving the effect of precreating the
specific named interpreter instance.

So as to make sure you don't think that that first interpreter
instance created in the parent and inherited by the child processes is
completely wasted, it should be pointed out that the first interpreter
instance created by Python is sort of special. In general it shouldn't
matter, but there is one case where it does. This is where a third
party extension module for Python has not been written so as to work
properly in a context where there are multiple sub interpreters.
Specifically, if a third party extension module used the simplified
API for GIL locking one can have problems using that module in
anything but the first interpreter instance created by Python. Thus,
the first instance is retained and in some cases it may be necessary
to force your application to run within the context of that
interpreter instance to get it to work where using such a module. If
you have to do this for multiple applications running under different
VirtualHost containers you loose your separation though, thus this is
only provided as a fallback when you don't have a choice.

I'll mention one other area in case you have the wrong idea about it
as well. In mod_python there is a feature for certain Python modules
to be reloaded. This feature is normally on by default but is always
recommended to be turned off in a production environment. To make it
quite clear, this feature does not mean that the modules which are
candidates for reloading will be reloaded on every request. Such
modules will only be reloaded if the code file for that module has
been changed. Ie., its modification time on disk has been changed. In
mod_python 3.3 where this feature is a bit more thorough and robust,
it will also reload a candidate module if some child or descendant of
the module has been changed.

So to summarise. Interpreter instances once created in the child
processes for a particular context are retained in memory and used
from one request to the next. Further, any modules loaded by code for
a request handler is retained in memory and do not have to be reloaded
on each request. Even when module reloading is enabled in mod_python,
modules are only reloaded where a code file associated with that
module has been changed on disk.

Does that clarify any misunderstandin gs you have?

So far it looks like the only problem that has been identified is one
that I already know about, which is that there isn't any good
documentation out there which describes how it all works. As a result
there are a lot of people out there who describe wrongly how they
think it works and thus give others the wrong impression. I already
knew this, as I quite often find myself having to correct statements
on various newgroups and in documentation for various Python web
frameworks. What is annoying is that even though you point out to some
of the Python web frameworks that what they state in their
documentation is wrong or misleading they don't correct it. Thus the
wrong information persists and keeps spreading the myth that there
must be some sort of problem where there isn't really. :-(

Graham

[Graham Dumpleton]

Re: Python does not play well with others

On Feb 6, 5:39 am, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:

John Nagle <n...@animats.c omwrites:

The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting.

>

Pre-forking doesn't reduce load; it just improves responsiveness.
You still pay for loading all the modules on every request. For
many AJAX apps, the loading cost tends to dominate the transaction.

>
I think the idea is that each pre-forked subprocess has its own
mod_python that services multiple requests serially.

And where 'worker' MPM is used, each child process can be handling
multiple concurrent requests at the same time. Similarly on Windows
although there is only one process.

New to me is the idea that you can have multiple separate Python
interpreters in a SINGLE process (mentioned in another post). I'd
thought that being limited to one interpreter per process was a
significant and hard-to-fix limitation of the current CPython
implementation that's unlikely to be fixed earlier than 3.0.

No such limitation exists with mod_python as it does all the
interpreter creation and management at the Python C API level. The one
interpreter per process limitation is only when using the standard
'python' runtime executable and you are doing everything in Python
code.

Graham

[sjdevnull@yahoo.com]

Re: Python does not play well with others

On Feb 5, 12:52 pm, John Nagle <n...@animats.c omwrote:

sjdevn...@yahoo .com wrote:

John Nagle wrote:

>

>Graham Dumpleton wrote:

>

>>On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:

>

>>>"Paul Boddie" <p...@boddie.or g.ukwrites:
Realistically, mod_python is a dead end for large servers,
>because Python isn't really multi-threaded. The Global Python
>Lock means that a multi-core CPU won't help performance.

>

The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting.

>
Pre-forking doesn't reduce load; it just improves responsiveness.
You still pay for loading all the modules on every request.

No, you don't. Each server is persistent and serves many requests--
it's not at all like CGI, and it reuses the loaded Python image.

So if you have, say, an expensive to load Python module, that will
only be executed once for each server you start...e.g. if you have
Apache configured to accept up to 50 connections, the module will be
run at most 50 times; once each of the 50 processes has started up,
they stick around until you restart Apache, unless you've configured
apache to only serve X requests in one process before restarting it.
(The one major feature that mod_python _is_ missing is the ability to
do some setup in the Python module prior to forking. That would make
restarting Apache somewhat nicer).

The major advantage of pre-forking is that you have memory protection
between servers, so a bug in one won't take down the whole apache
server (just the connection(s) that are affected by that bug). Most
shared hosting providers use pre-forking just for these stability
reasons.

A nice side effect of the memory protection is that you have
completely seperate Python interpreters in each process--while each
one is reused between connections, they run in independent processes
and the GIL doesn't come into play at all.

[Paul Rubin]

Re: Python does not play well with others

"Graham Dumpleton" <grahamd@dscpl. com.auwrites:

No such limitation exists with mod_python as it does all the
interpreter creation and management at the Python C API level. The one
interpreter per process limitation is only when using the standard
'python' runtime executable and you are doing everything in Python code.

Oh cool, I thought CPython used global and/or static variables or had
other obstacles to supporting multiple interpreters. Is there a
separate memory pool for each interpreter when you have multiple ones?
So each one has its own copies of 0,1,2,None,..., etc? How big is the
memory footprint per interpreter? I guess there's no way to timeshare
(i.e. with microthreads) between interpreters but that's o.k.

[Paul Rubin]

Re: Python does not play well with others

"Graham Dumpleton" <grahamd@dscpl. com.auwrites:

Yes, these per VirtualHost interpreter instances will only be created
on demand in the child process when a request arrives which
necessitates it be created and so there is some first time setup for
that specific interpreter instance at that point, but the main Python
initialisation has already occurred so this is minor.

Well ok, but what if each of those interpreters wants to load, say,
the cookie module? Do you have separate copies of the cookie module
in each interpreter? Does each one take the overhead of loading the
cookie module? It would be neat if there was a way of including
frequently used modules in the shared text segment of the
interpreters, as created during the initial build process. GNU Emacs
used to do something like that with a contraption called "unexec" (it
could dump out parts of its data segment into a pure (shared)
executable that you could then run without the overhead of loading all
those modules) but the capability went away as computers got faster
and it became less common to have a lot of Emacs instances weighing
down timesharing systems. Maybe it's time for a revival of those
techniques.

[Graham Dumpleton]

Re: Python does not play well with others

On Feb 6, 8:57 am, "sjdevn...@yaho o.com" <sjdevn...@yaho o.comwrote:

On Feb 5, 12:52 pm, John Nagle <n...@animats.c omwrote:
>
>
>

sjdevn...@yahoo .com wrote:

John Nagle wrote:

>

>>Graham Dumpleton wrote:

>

>>>On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:

>

>>>>"Paul Boddie" <p...@boddie.or g.ukwrites:
> Realistically,m od_pythonis a dead end for large servers,
>>because Python isn't really multi-threaded. The Global Python
>>Lock means that a multi-core CPU won't help performance.

>

The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting.

>

Pre-forking doesn't reduce load; it just improves responsiveness.
You still pay for loading all the modules on every request.

>
No, you don't. Each server is persistent and serves many requests--
it's not at all like CGI, and it reuses the loaded Python image.
>
So if you have, say, an expensive to load Python module, that will
only be executed once for each server you start...e.g. if you have
Apache configured to accept up to 50 connections, the module will be
run at most 50 times; once each of the 50 processes has started up,
they stick around until you restart Apache, unless you've configured
apache to only serve X requests in one process before restarting it.
(The one major feature thatmod_python_ is_ missing is the ability to
do some setup in the Python module prior to forking. That would make
restarting Apache somewhat nicer).

There would be a few issues with preloading modules before the main
Apache child process performed the fork.

The first is whether it would be possible for code to be run with
elevated privileges given that the main Apache process usually is
started as root. I'm not sure at what point it switches to the special
user Apache generally runs as and whether in the main process the way
this switch is done is enough to prevent code getting back root
privileges in some way, so would need to be looked into.

The second issue is that there can be multiple Python interpreters
ultimately created depending on how URLs are mapped, thus it isn't
just an issue with loading a module once, you would need to create all
the interpreters you think might need it and preload it into each. All
this will blow out the memory size of the main Apache process.

There is also much more possibility for code, if it runs up extra
threads, to interfere with the operation of the Apache parent process.
One particular area which could be a problem is where Apache wants to
do a restart, as it will attempt to unload the mod_python module and
reload it. Right now this may not be an issue as mod_python does the
wrong thing and doesn't shutdown Python allowing it to be
reinitialised when mod_python is reloaded, but in mod_wsgi (when
mod_python isn't also being loaded), it will shutdown Python. If there
is user code executing in a thread within the parent process this may
actually stop mod_wsgi from cleanly shutting down Python thus causing
Apache to hang.

All up, the risks of loading extra modules in the parent process
aren't worth it and could just result in things being less stable.

Graham