Improving interpreter startup speed

[David Cournapeau]

Re: Improving interpreter startup speed

On Mon, Oct 27, 2008 at 2:36 PM, Terry Reedy <tjreedy@udel.e duwrote:

>
It this a theoretical problem or an actual one, that we might have other
suggestions for?

Any command line based on python is a real example of that problem.
There are plenty of them.

David

[James Mills]

Re: Improving interpreter startup speed

On Mon, Oct 27, 2008 at 3:36 PM, Terry Reedy <tjreedy@udel.e duwrote:

It this a theoretical problem or an actual one, that we might have other
suggestions for?

Heaven knows! I hardly think invoking hundreds
and possibly thousands of short-lived python
interpreters to be an optimal solution that may
have spawned this particular thread.

--JamesMills

--
--
-- "Problems are solved by method"

[James Mills]

Re: Improving interpreter startup speed

On Mon, Oct 27, 2008 at 5:28 PM, David Cournapeau <cournape@gmail .comwrote:

Any command line based on python is a real example of that problem.
There are plenty of them.

Yes, but in most cases you are not invoking your
command-line app x times per y units of time.

--JamesMills

--
--
-- "Problems are solved by method"

[Paul Rubin]

Re: Improving interpreter startup speed

"James Mills" <prologic@short circuit.net.auw rites:

Heaven knows! I hardly think invoking hundreds
and possibly thousands of short-lived python
interpreters to be an optimal solution that may
have spawned this particular thread.

It's not optimal but it is very common (CGI for example).

[David Cournapeau]

Re: Improving interpreter startup speed

On Mon, Oct 27, 2008 at 4:33 PM, James Mills
<prologic@short circuit.net.auw rote:

Yes, but in most cases you are not invoking your
command-line app x times per y units of time.

Depends on the tool: build tool and source control tools are example
it matters (specially when you start interfaciing them with IDE or
editors). Having fast command line tools is an important feature of
UNIX, and if you want to insert a python-based tool in a given
pipeline, it can hurt it the pipeline is regularly updated.

cheers,

David

[James Mills]

Re: Improving interpreter startup speed

On Mon, Oct 27, 2008 at 5:36 PM, Paul Rubin
<"http://phr.cx"@nospam. invalidwrote:

It's not optimal but it is very common (CGI for example).

Which is why we (The Python Community)
created WSGI and mod_wsgi. C"mon guys
these "problems" are a bit old and out
dated :)

--JamesMills

--
--
-- "Problems are solved by method"

[Gabriel Genellina]

Re: Improving interpreter startup speed

En Sun, 26 Oct 2008 23:52:32 -0200, James Mills
<prologic@short circuit.net.aue scribió:

On Mon, Oct 27, 2008 at 4:12 AM, Benjamin Kaplan
<benjamin.kapla n@case.eduwrote :

>You must be in a real big hurry if half a second matters that much to
>you.
>Maybe if it took 5 seconds for the interpreter to start up, I could
>understand having a problem with the start up time.

>
+1 This thread is stupid and pointless.
Even for a so-called cold startup 0.5s is fast enough!

I don't see the need to be rude.
And I DO care for Python startup time and memory footprint, and others do
too. Even if it's a stupid thing (for you).

--
Gabriel Genellina

[James Mills]

Re: Improving interpreter startup speed

On Mon, Oct 27, 2008 at 5:40 PM, David Cournapeau <cournape@gmail .comwrote:

Depends on the tool: build tool and source control tools are example
it matters (specially when you start interfaciing them with IDE or
editors). Having fast command line tools is an important feature of
UNIX, and if you want to insert a python-based tool in a given
pipeline, it can hurt it the pipeline is regularly updated.

Fair enough. But still:
0.5s old startup is fast enough
0.08s warm startup is fast enough.

Often "fast enough" is "fast enough"

--JamesMills

--
--
-- "Problems are solved by method"

[James Mills]

Re: Improving interpreter startup speed

On Mon, Oct 27, 2008 at 5:46 PM, Gabriel Genellina
<gagsl-py2@yahoo.com.a rwrote:

>+1 This thread is stupid and pointless.
>Even for a so-called cold startup 0.5s is fast enough!

>
I don't see the need to be rude.
And I DO care for Python startup time and memory footprint, and others do
too. Even if it's a stupid thing (for you).

I apologize. I do not see the point comparing Python with
RUby however, or Python with anything else.

So instead of coming up with arbitary problems, why don't
we come up with solutions for "Improving Interpreter Startup Speeds" ?

I have only found that using the -S option speeds it up
significantly, but that's only if you're not using any site
packages and only using the built in libraries.

Can site.py be improved ?

--JamesMills

--
--
-- "Problems are solved by method"

[durumdara@gmail.com]

Re: Improving interpreter startup speed

To make faster python, you can do:

1.) Use mod_python, and not cgi.
2.) Use other special python server that remaining in memory, and call
it from compiled C code. For example, the C code communicate this server
with pipes, tcp, (or with special files, and the result will come back
in other file).
You can improve this server when you split threads to python
subprocesses, and they still alive for X minutes.
You have one control process (py), and this (like the apache)
communicate the subprocesses, kill them after timeout, and start a new
if needed.

dd

James Mills írta:

On Mon, Oct 27, 2008 at 5:46 PM, Gabriel Genellina
<gagsl-py2@yahoo.com.a rwrote:

>>+1 This thread is stupid and pointless.
>>Even for a so-called cold startup 0.5s is fast enough!

>I don't see the need to be rude.
>And I DO care for Python startup time and memory footprint, and others do
>too. Even if it's a stupid thing (for you).

>
I apologize. I do not see the point comparing Python with
RUby however, or Python with anything else.
>
So instead of coming up with arbitary problems, why don't
we come up with solutions for "Improving Interpreter Startup Speeds" ?
>
I have only found that using the -S option speeds it up
significantly, but that's only if you're not using any site
packages and only using the built in libraries.
>
Can site.py be improved ?
>
--JamesMills
>

[Terry Reedy]

Re: Improving interpreter startup speed

David Cournapeau wrote:

On Mon, Oct 27, 2008 at 2:36 PM, Terry Reedy <tjreedy@udel.e duwrote:

>It this a theoretical problem or an actual one, that we might have other
>suggestions for?

>
Any command line based on python is a real example of that problem.

No it is not.
The specific problem that you wrote and I responded to was

"Not if the startup is the main cost for a command you need to repeat
many times. "

in a short enough period so that the startup overhead was a significant
fraction of the total time and therefore a burden.

[Terry Reedy]

Re: Improving interpreter startup speed

James Mills wrote:

So instead of coming up with arbitary problems, why don't
we come up with solutions for "Improving Interpreter Startup Speeds" ?

The current developers, most of whom use Python daily, are aware that
faster startup would be better. 2.6 and 3.0 start up quicker because
the some devs combed through the list of startup imports to see what
could be removed (or, in one case, I believe, consolidated). Some were.
Anyone who is still itching on this subject can seek further
improvements and, if successful, submit a patch.

Or, one could check the Python wiki for a StartUpTime page and see if
one needs to be added or improved/updated with information from the
PyDev list archives to make it easier for a new developer to get up to
speed on what has already been done in this area and what might be done.

tjr

[Lie]

Re: Improving interpreter startup speed

On Oct 27, 2:36 pm, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:

"James Mills" <prolo...@short circuit.net.auw rites:

Heaven knows! I hardly think invoking hundreds
and possibly thousands of short-lived python
interpreters to be an optimal solution that may
have spawned this particular thread.

>
It's not optimal but it is very common (CGI for example).

CGI? When you're talking about CGI, network traffic is simply the
biggest bottleneck, not something like python interpreter startup
time. Also, welcome to the 21st century, where CGI is considered as an
outdated protocol.

[James Mills]

Re: Improving interpreter startup speed

On Wed, Oct 29, 2008 at 9:43 PM, Lie <Lie.1296@gmail .comwrote:

On Oct 27, 2:36 pm, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:

>It's not optimal but it is very common (CGI for example).

>
CGI? When you're talking about CGI, network traffic is simply the
biggest bottleneck, not something like python interpreter startup
time. Also, welcome to the 21st century, where CGI is considered as an
outdated protocol.

That's right. That's why we have WSGI. That's
why we built mod_wsgi for Apache. Hell taht's
why we actually have really nice Web Frameworks
such as: CherryPy, Pylons, Paste, etc. They
perform pretty damn well!

--JamesMills

--
--
-- "Problems are solved by method"

[J. Clifford Dyer]

Re: Improving interpreter startup speed

Maybe Ruby is the right language for your need.

Just sayin'.



On Sun, 2008-10-26 at 13:19 +0000, Pedro Borges wrote:

The scripts i need to run but be executed with no apparent delay
specially when the text transforms are simple.
>
>
On Oct 26, 2008, at 11:13 AM, James Mills wrote:
>

On Sun, Oct 26, 2008 at 11:23 AM, BJörn Lindqvist
<bjourne@gmail. comwrote:

How are you getting those numbers? 330 μs is still pretty fast,
isn't
it? :) Most disks have a seek time of 10-20 ms so it seem implausible
to me that Ruby would be able to cold start in 47 ms.

$ time python -c "pass"

real 0m0.051s
user 0m0.036s
sys 0m0.008s

$ time python3.0 -c "pass"

real 0m0.063s
user 0m0.048s
sys 0m0.004s

And yes I agree. the CPython interpreter startup times is
a stupid thing to be worrying about, especially since that
is never the bottleneck.

Python loads plenty fast enough!

--JamesMills

--
--
-- "Problems are solved by method"

>
--
http://mail.python.org/mailman/listinfo/python-list