Overhead of individual python apps

[Qopit]

I'm setting up a system that consists of several small python
applications that all communicate amongst each other on the same pc.

When running in Windows, launching each application generates a
process, and each of those processes ends up taking up > 4MB of system
memory. This memory usage is as reported by the Windows Task manager
for the python.exe image name.

My Question: Is there any way to reduce this per-process overhead? eg:
can you set it somehow so that one python.exe instance handles multiple
processes?

One possibility considered is to run them as threads of a single
process rather than multiple processes, but this has other drawbacks
for my application and I'd rather not,

Another possibility I considered is to strip out all but the most
essential imports in each app, but I tested this out and it has
marginal benefits. I demonstrated to myself that a simple one liner
app consisting of 'x = raw_input()' still eats up > 2.7MB .

I also tried -O but it, not surprisingly, did nothing for the
one-liner.

I'm simply running the .py files and I am still on v2.3

All help appreciated!

Thanks,
Russ

[Mike Meyer]

Re: Overhead of individual python apps

"Qopit" <russandheather @gmail.com> writes:
[color=blue]
> When running in Windows, launching each application generates a
> process, and each of those processes ends up taking up > 4MB of system
> memory. This memory usage is as reported by the Windows Task manager
> for the python.exe image name.[/color]

The first step is to clarify what's being reported. If WTM is
reporting the total memory usage for each process, then it's over
estimating the total usage by a considerable amount. In particular,
all the Python executable code should be shared by all the
processes. Unless you load compiled extensions, anyway - those will
only be shared by the ones that use them. You'll need something that
will give you the stack, heap and code segment sizes separately to
work out what the memory usage really is.
[color=blue]
> My Question: Is there any way to reduce this per-process overhead? eg:
> can you set it somehow so that one python.exe instance handles multiple
> processes?[/color]

The OS should do that for you by default.
[color=blue]
> One possibility considered is to run them as threads of a single
> process rather than multiple processes, but this has other drawbacks
> for my application and I'd rather not,[/color]

That shouldn't help memory usage - the data that isn't across
processes would need to be thread-private in any case.

The reason for the uncertainty is that I'm not positive that Windows
behaves sanely in this area. It may be that Windows doesn't have
shared executables. In this case, one solution is to move to a modern
OS - like v6 Unix :-).

<mike
--
Mike Meyer <mwm@mired.or g> http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

[Larry Bates]

Re: Overhead of individual python apps

Several apps using 4Mb each shouldn't be very much
memory (maybe 20Mb at most). You didn't say how
much memory was in your machine, but 256Mb of memory
will cost you no more than $50. Not really worth a
lot of effort.

-Larry

Qopit wrote:[color=blue]
> I'm setting up a system that consists of several small python
> applications that all communicate amongst each other on the same pc.
>
> When running in Windows, launching each application generates a
> process, and each of those processes ends up taking up > 4MB of system
> memory. This memory usage is as reported by the Windows Task manager
> for the python.exe image name.
>
> My Question: Is there any way to reduce this per-process overhead? eg:
> can you set it somehow so that one python.exe instance handles multiple
> processes?
>
> One possibility considered is to run them as threads of a single
> process rather than multiple processes, but this has other drawbacks
> for my application and I'd rather not,
>
> Another possibility I considered is to strip out all but the most
> essential imports in each app, but I tested this out and it has
> marginal benefits. I demonstrated to myself that a simple one liner
> app consisting of 'x = raw_input()' still eats up > 2.7MB .
>
> I also tried -O but it, not surprisingly, did nothing for the
> one-liner.
>
> I'm simply running the .py files and I am still on v2.3
>
> All help appreciated!
>
> Thanks,
> Russ
>[/color]

[Paul Rubin]

Re: Overhead of individual python apps

Larry Bates <larry.bates@we bsafe.com> writes:[color=blue]
> Several apps using 4Mb each shouldn't be very much memory (maybe
> 20Mb at most). You didn't say how much memory was in your machine,
> but 256Mb of memory will cost you no more than $50. Not really
> worth a lot of effort.[/color]

That is bogus reasoning. I can't add 256mb to the machine I'm typing
on right now for any amount of money, let alone $50. It's an older
laptop, it's maxed out at 512 meg, and I can't add any more to it,
period. An even older laptop that I still use from time to time is
maxed out at 72 meg. And my Sharp Zaurus (Linux handheld) has 32 meg
and is not expandable. If Python claims to be a lightweight language,
it should not present any obstacles to running on machines of that
class. It's already slightly annoying that Python can't run well in
an 8 meg Palm Pilot, since there were many Common Lisp implementations
that ran tolerably in less memory than that.

[ncf]

Re: Overhead of individual python apps

First, please don't get so upset at people's replies (if you weren't
upset, that's how it was coming across, so my apologies). No matter
what newsgroup/forum you go to, there's always someone who's going to
suggest something like that.

Anyways, I'm fairly certain there are some minimalistic Python variants
designed for older computers and such. Maybe something to check into?

HTH,
-Wes

[Dennis Lee Bieber]

Re: Overhead of individual python apps

On 27 Sep 2005 15:11:57 -0700, "Qopit" <russandheather @gmail.com>
declaimed the following in comp.lang.pytho n:
[color=blue]
> When running in Windows, launching each application generates a
> process, and each of those processes ends up taking up > 4MB of system
> memory. This memory usage is as reported by the Windows Task manager
> for the python.exe image name.
>[/color]
Ah, but is that physical memory consumed, or virtual memory MAPPED
to the processes.

Most of Python is a DLL; as I recall, most DLLs are shared -- just
the process local data needs to be allocated.
--[color=blue]
> =============== =============== =============== =============== == <
> wlfraed@ix.netc om.com | Wulfraed Dennis Lee Bieber KD6MOG <
> wulfraed@dm.net | Bestiaria Support Staff <
> =============== =============== =============== =============== == <
> Home Page: <http://www.dm.net/~wulfraed/> <
> Overflow Page: <http://wlfraed.home.ne tcom.com/> <[/color]

[Fredrik Lundh]

Re: Overhead of individual python apps

Paul Rubin wrote:
[color=blue][color=green]
>> Several apps using 4Mb each shouldn't be very much memory (maybe
>> 20Mb at most). You didn't say how much memory was in your machine,
>> but 256Mb of memory will cost you no more than $50. Not really
>> worth a lot of effort.[/color]
>
> That is bogus reasoning.[/color]

not if you're a professional software developer and someone's paying you
to develop an application that is to be run on a platform that they control.

people are expensive, but hardware is extremely cheap these days.

</F>

[Paul Rubin]

Re: Overhead of individual python apps

"Fredrik Lundh" <fredrik@python ware.com> writes:[color=blue][color=green]
> > That is bogus reasoning.[/color]
>
> not if you're a professional software developer and someone's paying you
> to develop an application that is to be run on a platform that they control.[/color]

An awful lot of Python targeted users are not in that situation, so if
Python's usability suffers for them when it doesn't have to, then
something is wrong with Python.

[Fredrik Lundh]

Re: Overhead of individual python apps

Paul Rubin wrote:
[color=blue]
> An awful lot of Python targeted users are not in that situation, so if
> Python's usability suffers for them when it doesn't have to, then
> something is wrong with Python.[/color]

(and so we go from the OP:s "I'm setting up a system" to the usual c.l.python "but
I can come up with another example" crap...)

But back to the "memory measurements" and your claim that "something is wrong
with Python [if the task manager shows large numbers]"...

Have you looked at a task manager on a lightly loaded W2K or WXP system lately?
On this PC, I currently have a couple of browser windows open, a newsreader, and
a few other programs. The only real app I have that uses less than 4 megabytes
*according to the task manager* is a 2.5 megabyte putty terminal window; most
apps are in the 15-50 megabyte range.

Even a trivial operation, like opening another Firefox tab, can make that process
grow by more than 4 megabytes, and if you browse around a little, you can easily
end up with 30-50 megabytes in "mem usage" -- until you minimize the browser,
and find that Firefox only uses 1.5 megabytes. Maximize it again, and it uses 12
megabytes.

Play the same games with a Python interpreter, and you'll find that the task manager
may report anything from just over 50 kilobytes to just under 5 megabytes.

Here's a random google link that discusses the task manager figures in a little more
detail:

http://www.itwriting.com/dotnetmem.php

"Newcomer to .NET says: I've just been looking at Task Manager. Why
does my simple VB.NET Windows application use 12MB RAM?"

"Don't worry. The app doesn't really use that much RAM."

(for python, the "private" memory use is usually ~1.5 megabytes for a "empty" 2.4
process, and some of that will only occupy space in the paging file... for firefox with
a simple page loaded into a single tab, the private space is ~10 megabytes)

</F>

[Steve Holden]

Re: Overhead of individual python apps

Paul Rubin wrote:[color=blue]
> "Fredrik Lundh" <fredrik@python ware.com> writes:
>[color=green][color=darkred]
>>>That is bogus reasoning.[/color]
>>
>>not if you're a professional software developer and someone's paying you
>>to develop an application that is to be run on a platform that they control.[/color]
>
>
> An awful lot of Python targeted users are not in that situation, so if
> Python's usability suffers for them when it doesn't have to, then
> something is wrong with Python.[/color]

If a useful subset of Python can be crammed into a Nokia cell phone then
I really don't think there's much to complain about (except that "it
hasn't been done for *my* machine").

Even embedded systems are much larger now than the minicomputers of
yesteryear. Everything's relative. Just wait three years!

i-remember-when-we-'ad-ter-code-seven-kilobytes-of-assembly-language-wi'-nowt-bu'-a-teletype-ter-edit-t'-paper-tape-wi'-ly
y'rs - steve

--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.pycon.org

[Paul Boddie]

Re: Overhead of individual python apps

Steve Holden wrote:[color=blue]
> Even embedded systems are much larger now than the minicomputers of
> yesteryear. Everything's relative. Just wait three years![/color]

Had you placed such a bet in 2000, you'd have cleaned up at the
"Moore's Law Casino", but there are various factors at work now which
complicate the once-inevitable trends of hardware performance and the
corresponding advice to people wishing to speed up or slim down their
software: the flattening out of the CPU frequency curve and the
tendency of CPU manufacturers to choose multiple core strategies are
two things which prevent applications from speeding up all by
themselves; whilst storage density is still increasing, as far as I
know, I'd imagine other strategies being adopted in system architecture
which could make performance tradeoffs more pronounced in accessing all
that storage.

In other words, expect many more threads about global interpreter locks
in the coming three years than we've seen in the last three or even six
years. ;-)

Paul

[Steve Holden]

Re: Overhead of individual python apps

Paul Boddie wrote:[color=blue]
> Steve Holden wrote:
>[color=green]
>>Even embedded systems are much larger now than the minicomputers of
>>yesteryear. Everything's relative. Just wait three years![/color]
>
>
> Had you placed such a bet in 2000, you'd have cleaned up at the
> "Moore's Law Casino", but there are various factors at work now which
> complicate the once-inevitable trends of hardware performance and the
> corresponding advice to people wishing to speed up or slim down their
> software: the flattening out of the CPU frequency curve and the
> tendency of CPU manufacturers to choose multiple core strategies are
> two things which prevent applications from speeding up all by
> themselves; whilst storage density is still increasing, as far as I
> know, I'd imagine other strategies being adopted in system architecture
> which could make performance tradeoffs more pronounced in accessing all
> that storage.
>
> In other words, expect many more threads about global interpreter locks
> in the coming three years than we've seen in the last three or even six
> years. ;-)
>[/color]
I seem to remember blogging about this a while ago, or perhaps just
ranting at c.l.py ... anyway, bottom line is that most people have more
than adequate power for their current needs in a present-day
single-processor computer.

Expect also to see computational methods focus more on memory-intensive
techniques impractical in the past.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.pycon.org

[Qopit]

Re: Overhead of individual python apps

Thanks - this is all very interesting...
[color=blue]
> Ah, but is that physical memory consumed, or virtual memory MAPPED
> to the processes.[/color]

and
[color=blue]
> for python, the "private" memory use is usually ~1.5 megabytes for a "empty" 2.4
> process, and some of that will only occupy space in the paging file... for firefox with
> a simple page loaded into a single tab, the private space is ~10 megabytes)[/color]

and most useful:
[color=blue]
> http://www.itwriting.com/dotnetmem.php[/color]

I had no idea that the memory usage reported by Windows Task Manager
(WTM) was so different than what I expected it would be. It doesn't
seem terribly useful to me right now.

After looking into that link (and discovering the perfmon app... can't
believe I never knew about such an amazingly useful tool!) below are
the results of some memory reporting checks that I ran on the simple
one-liner app (x = raw_input()). The "On Launch" is what was reported
immediately after launching the app, and the "Window Min'd" is
following a simple minimization for the window.

Memory report: On Launch Window Min'd
-------------- --------- ------------
WTM Mem Usage: 2,756K 88K
PerfMon Private bytes: 1,540,096 1,540,096
PerfMon Working set: 2,822,144 90,112 **

Basically it looks like the privately allocated memory for a 2.3 app
that cannot be shared by other processes (close to what I want,
neglecting paging possibilities) is the 1.5 MB that Fredrik reported.
Presumably using perfmon as well?

Maybe I'm a stickler, but this still seems pretty high for each and
every application to need to privately hold. Am I still misreading
this somehow? If not, I'd still love for this to be smaller... is
there any way to reduce this further?

Russ

** There was an interesting transient in the Working set during
minimization all the way up to 9 MB... presumably this is because
during minimization the "set of memory pages touched recently" jumps up
because of the transactions needed for the minimization.

[Peter Hansen]

Re: Overhead of individual python apps

Fredrik Lundh wrote:[color=blue]
> (for python, the "private" memory use is usually ~1.5 megabytes for a "empty" 2.4
> process, and some of that will only occupy space in the paging file... for firefox with
> a simple page loaded into a single tab, the private space is ~10 megabytes)[/color]

I believe a relatively simple, if crude, way of measuring this sort of
thing (at least for console programs) is to open a handful of command
prompts, check the Physical Memory Available measurement in the Task
Manager, and then load one instance of the program in each console,
checking the drop in physical memory available at each step. After
three or four instances are loaded, the approximate real memory usage
for each should be readily apparent.

Obviously there are a dozen issues that can screw with this, including
having too little memory free at all, and having other programs open
which are very actively allocating or freeing memory.

It does seem to back up the "~1.5MB" number above, and certainly shows
that the total is nowhere near 4MB per Python interpreter.

-Peter

[Dennis Lee Bieber]

Re: Overhead of individual python apps

On 28 Sep 2005 07:46:23 -0700, "Qopit" <russandheather @gmail.com>
declaimed the following in comp.lang.pytho n:
[color=blue]
>
> Maybe I'm a stickler, but this still seems pretty high for each and
> every application to need to privately hold. Am I still misreading
> this somehow? If not, I'd still love for this to be smaller... is
> there any way to reduce this further?
>[/color]
I've not investigated -- but what is the default stack size given to
an application?
--[color=blue]
> =============== =============== =============== =============== == <
> wlfraed@ix.netc om.com | Wulfraed Dennis Lee Bieber KD6MOG <
> wulfraed@dm.net | Bestiaria Support Staff <
> =============== =============== =============== =============== == <
> Home Page: <http://www.dm.net/~wulfraed/> <
> Overflow Page: <http://wlfraed.home.ne tcom.com/> <[/color]