Python compilers?

[Svein Ove Aas]

Re: Python compilers?

Peter Hansen wrote:
[color=blue]
> Fuzzyman wrote:
>[color=green]
>> Funnily though, most people call Java a compiled language, but it only
>> compiles to Java bytecode which runs on the virtual machine. Python
>> precompiles to python bytecode which runs on the python virtual
>> machine. So arguably it is *as* compiled as Java.....[/color]
>
> Actually, there are compilers that produce native machine code from
> Java for several CPUs available, and they are used at least in the
> embedded world.
>[/color]
There is also GCJ as part of the GCC, which can compile both .class
and .java files. Its libraries aren't complete yet, but I'm sure it's
only a matter of time.

[Carl Banks]

Re: Python compilers?

Heather Coppersmith wrote:[color=blue]
>
>
> On Fri, 21 May 2004 03:03:51 GMT,
> Carl Banks <imbosol@aerojo ckey.invalid> wrote:
>[color=green]
>> Paul Rubin wrote:[color=darkred]
>>>
>>>
>>> Carl Banks <imbosol@aerojo ckey.invalid> writes:
>>>> > The example above kills any attempt to turn a.bar() into a static
>>>> > procedure call.
>>>>
>>>> Of course it does--but it's one method. A compiler, if it's good,
>>>> would only make the optization on methods named "bar", and it could
>>>> probably pare the number of possible classes it could happen to down
>>>> to only a few.
>>>
>>> How could it possibly know? The reassignment of a.bar could happen
>>> anytime, anywhere in the code. Maybe even in an eval.[/color][/color]
>[color=green]
>> And if it happens anytime, anywhere in the code, the compiler will see
>> it and create more general code. Or is that impossible?[/color]
>
> The compiler might not see it. Any function/method/module called while
> a is in scope might change a.bar. It doesn't even take a perverted
> introspection tool:
>
> module foo:
>
> import bar
> class C:
> pass
> a = C( )
> a.bar = 'bar'
> bar.bar( a )
> a.bar = a.bar + 'bar' # which 'add' instruction should this compile to?
>
> module bar:
>
> def bar( x ):
> x.bar = 3[/color]


Compiler builds a big call-tree. Compiler sees that the object "a" is
passed to a function that might rebind bar. Compiler thinks to
itself, "better mark C.bar as a dynamic attribute." Compiler sees
dynamic attirbute and addition. Compiler generates generic add code.

You could have all kinds of setattrs, evals, and unanalysable data
structures (a la pickle), that could turn the code into a nightmare
that a compiler couldn't do anything with. But surely, guarding
against all of that, a good enough static compiler can still reduce a
lot of good, maintainable Python code to it's static case.


And, frankly, I still don't see how this is different from Lisp.

(defun foo ()
(let ((x 1))
(setq x (bar x))
(setq x (+ x 1))))

In another package:

(defun bar (x)
#C(2.0 1.0))

If I recall, + can work on ints, floats, bignums, rationals, and
complex numbers, at least. What one instruction does + compile to
here?

[color=blue][color=green]
>> As for eval, well it would be silly to even allow eval in a compiled
>> program; kind of defeats the purpose of compling. You might let it
>> run in its own namespace so it can only affect certain objects.[/color]
>
> I reiterate my comments regarding type declarations, add new comments
> regarding backwards compatibility, and note that Lisp allows eval in
> compiled code.[/color]

It's highly frowned upon cause it interferes with everything it
touches.


--
CARL BANKS http://www.aerojockey.com/software
"If you believe in yourself, drink your school, stay on drugs, and
don't do milk, you can get work."
-- Parody of Mr. T from a Robert Smigel Cartoon

[Andrew MacIntyre]

Re: Python compilers?

On Fri, 21 May 2004, Svein Ove Aas wrote:
[color=blue]
> Peter Hansen wrote:
>[color=green]
> > Actually, there are compilers that produce native machine code from
> > Java for several CPUs available, and they are used at least in the
> > embedded world.[/color]
>
> There is also GCJ as part of the GCC, which can compile both .class
> and .java files. Its libraries aren't complete yet, but I'm sure it's
> only a matter of time.[/color]

Hmmm... anyone tried GCJ on Jython?

--
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andymac@bullsey e.apana.org.au (pref) | Snail: PO Box 370
andymac@pcug.or g.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

[Paul Rubin]

Re: Python compilers?

Carl Banks <imbosol@aerojo ckey.invalid> writes:[color=blue]
> If I recall, + can work on ints, floats, bignums, rationals, and
> complex numbers, at least. What one instruction does + compile to
> here?[/color]

Lisp supports type declarations which advise the compiler in those
situations. A few such proposals have been made for Python, but none
have taken off so far.

[Carl Banks]

Re: Python compilers?

Paul Rubin wrote:[color=blue]
> Carl Banks <imbosol@aerojo ckey.invalid> writes:[color=green]
>> If I recall, + can work on ints, floats, bignums, rationals, and
>> complex numbers, at least. What one instruction does + compile to
>> here?[/color]
>
> Lisp supports type declarations which advise the compiler in those
> situations. A few such proposals have been made for Python, but none
> have taken off so far.[/color]

Yes, that's been established. There's two questions remaining for me:

1. These claims that Lisp code can approach 50 percent the speed of C,
is that with or without the optional type declarations?

2. If you don't use the declarations, does compiling Lisp help? If it
does (and nothing I've read indicated that is doesn't), it
definitely casts some doubt on the claim that compiling Python
wouldn't help. That's kind of been my point all along.

I think (and I'm wrapping this up, cause I think I made my point)
compiling Python could help, even without type declarations, but
probably not as much as in Lisp. It could still make inferences or
educated guesses, like Lisp compilers do; just maybe not as often.


--
CARL BANKS http://www.aerojockey.com/software
"If you believe in yourself, drink your school, stay on drugs, and
don't do milk, you can get work."
-- Parody of Mr. T from a Robert Smigel Cartoon

[Ville Vainio]

Re: Python compilers?

>>>>> "Andrew" == Andrew MacIntyre <andymac@bullse ye.apana.org.au > writes:
[color=blue][color=green]
>> There is also GCJ as part of the GCC, which can compile both .class
>> and .java files. Its libraries aren't complete yet, but I'm sure it's
>> only a matter of time.[/color][/color]

Andrew> Hmmm... anyone tried GCJ on Jython?

Native code will not help much if the created native code is of type:

arg=lookup(obje ct1, "fooarg")
f = lookup(object2, "foomethod" )
call(f,arg)

For the performance that is expected of native code we need direct
dispatching with the addresses of the functions known at the compile
time, or via direct indexing of linear virtual tables.

(I'm speaking of static compilation here - what I said may not apply
to psyco)

--
Ville Vainio http://tinyurl.com/2prnb

[Heather Coppersmith]

Re: Python compilers?

On Sat, 22 May 2004 05:54:24 GMT,
Carl Banks <imbosol@aerojo ckey.invalid> wrote:
[color=blue]
> ... There's two questions remaining for me:[/color]
[color=blue]
> 1. These claims that Lisp code can approach 50 percent the speed
> of C, is that with or without the optional type declarations?[/color]

That would be *with* type declarations.
[color=blue]
> 2. If you don't use the declarations, does compiling Lisp help?
> If it does (and nothing I've read indicated that is doesn't),
> it definitely casts some doubt on the claim that compiling
> Python wouldn't help. That's kind of been my point all
> along.[/color]

In _Common Lisp The Language, Second Edition_, by Guy Steele
(CLTL2), on page 686, in section 25.1.3 ("Compilatio n
Environment") there is a lengthy list of assumptions the compiler
makes, including:

o ... within a named function, a recursive call to a function
of the same name refers to the same function [barring
notinline declarations] ...

o ... a call within the file being compiled to a named
function that is defined in that file refers to that
function [barring notinline declarations] ...

o ... the signature (or "interface contract") of all built-in
Common Lisp functions will not change ...

o ... the signature (or "interface contract") of functions
with ftype information will not change.

o ... any type definition made with defstruct or deftype in
the compile-type environment will retain the same definition
in the run-time environment. It may also assume that a
class defined in the compile-time environment will be
defined in the same run-time environment in such a way as to
have the same superclasses and metaclass ...

o ... if type declarations are present in the compile-time
environment, the corresponding variables and functions
present in the run-time environment will actually be of
those types ...

So compiled Lisp is a less dynamic than Python, and the built-in
functionality is large and guaranteed to match its textbook
definitions, which gives Lisp Compilers more than a fighting
chance. I'm pretty sure I've seen that summary on c.l.p before.
[color=blue]
> I think (and I'm wrapping this up, cause I think I made my
> point) compiling Python could help, even without type
> declarations, but probably not as much as in Lisp. It could
> still make inferences or educated guesses, like Lisp compilers
> do; just maybe not as often.[/color]

I agreem, with emphasis on "could" and some hesitation on how much
guessing I want my compiler to do.

Regards,
Heather

--
Heather Coppersmith
That's not right; that's not even wrong. -- Wolfgang Pauli

[Terry Reedy]

Re: Python compilers?


"Heather Coppersmith" <me@privacy.net > wrote in message
news:m23c5smwos .fsf@unique.pho ny.fqdn...[color=blue]
>
> In _Common Lisp The Language, Second Edition_, by Guy Steele
> (CLTL2), on page 686, in section 25.1.3 ("Compilatio n
> Environment") there is a lengthy list of assumptions the compiler
> makes, including:
>
> o ... within a named function, a recursive call to a function
> of the same name refers to the same function [barring
> notinline declarations] ...
>
> o ... a call within the file being compiled to a named
> function that is defined in that file refers to that
> function [barring notinline declarations] ...
>
> o ... the signature (or "interface contract") of all built-in
> Common Lisp functions will not change ...[/color]

Interesting. PyCode can be sped up by making the same assumptions.
Richard Hettinger's recent recipe implements these assumptions. (It was
rejected as part of the standard lib for being too implementation specific,
but remains available in the archives and Python Cookbook site.)

[snip less applicable to Python today stuff]
[color=blue]
> So compiled Lisp is a less dynamic than Python,[/color]

The balance between flexibity and speed continues to be debated by the
developers.

Terry J. Reedy

[Günter Jantzen]

Re: Python compilers?

"Terry Reedy" <tjreedy@udel.e du> schrieb im Newsbeitrag
news:mailman.18 3.1085238574.69 49.python-list@python.org ...[color=blue]
>[color=green]
> > So compiled Lisp is a less dynamic than Python,[/color]
>
> The balance between flexibity and speed continues to be debated by the
> developers.[/color]

And should be debated by the users, too.

Not all of Pythons dynamics is really necessary to write good programms.
Writing good software needs creativity and discipline. We like to work with
Python, because it allows us creativitity. Sometimes to enforce a grain of
discipline in Python would not be bad. There have been many discussions
about programming idioms and design patterns in the last years. They give
not so high level languages the ability to simulate high level features.
They could also help to give a very high level language more solid ground.
And they normalize the way to handle common classes of problems

"""There should be one-- and preferably only one --obvious way to do it.
"""
We could try to search patterns, idioms, antipatterns which could establish
a better way to use the Python language:
-- to avoid unnecessary dynamics
-- to avoid unnecessary features
-- to establish the 'obvious way to do it'
-- without loosing the power and expressiveness of python

"""In the face of ambiguity, refuse the temptation to guess.
"""
Possible sideeffect:.- it could be easier for mechanical tools (Pyrex,
Psycho, PyPy, Starkiller) to make Python faster

Not all people will agree about a useful set of patterns. But this is not
necessary now.
Patterns and idioms do not change a language immediately. They do not fall
from heaven, it is necessary to play around and to gain some experience.

I think patterns define a subset of a language and some constraints on
programms
Maybe this constrained subset (or subsets if there are controversal ideas)
can be formalized and be checked by 'pylintplus' or 'pythoncheckerX ' (do not
google).
Or there would be a configurable universal 'Monsterchecker '. In the first
line of our sources we expose a checkpolicy and the checker will follow this
policy.
If patterns can not be formalized so easy, they can be established by
convention.

I do not know the result of this process ...
Now some ideas what could be done in the near future

1) Integration of Pyrex into Python.
[strong variant]
Syntactically this languages are close. Maybe they could be changed so, that
every valid Pyrex source is also a valid Python source with -hopefully- the
same semantics. Special Pyrex tokens would be allowed, but meaningless when
used in the Python context.
Benefit: more homogenous environment. Little changes in the program could be
made without a C-Compilation step.
Maybe the support of interfaces would be necessary in Python, but this is
not a bad idea.
Then rewrite some parts of the standard library in Pyrex

[weak variant]
Rewrite some parts of the standard library in Pyrex

2) Find a set of constraints which make it easier for Psyco (now) and
Starkiller (later) to do their job

3) Implement Monsterchecker modes for 1 and 2

Regards
Guenter

[Andrew MacIntyre]

Re: Python compilers?

On Sat, 22 May 2004, Ville Vainio wrote:
[color=blue][color=green][color=darkred]
> >>>>> "Andrew" == Andrew MacIntyre <andymac@bullse ye.apana.org.au > writes:[/color][/color]
>[color=green][color=darkred]
> >> There is also GCJ as part of the GCC, which can compile both .class
> >> and .java files. Its libraries aren't complete yet, but I'm sure it's
> >> only a matter of time.[/color][/color]
>
> Andrew> Hmmm... anyone tried GCJ on Jython?
>
> Native code will not help much if the created native code is of type:
>
> arg=lookup(obje ct1, "fooarg")
> f = lookup(object2, "foomethod" )
> call(f,arg)
>
> For the performance that is expected of native code we need direct
> dispatching with the addresses of the functions known at the compile
> time, or via direct indexing of linear virtual tables.
>
> (I'm speaking of static compilation here - what I said may not apply
> to psyco)[/color]

I was asking more out of interest in the potential for creating
distributable binaries (which is usually one of the desires of people
looking for compilers), than performance specifically. Of course it
would be nice if a GCJ compiled Jython app could at least match the
performance of CPython for the same app.

--
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andymac@bullsey e.apana.org.au (pref) | Snail: PO Box 370
andymac@pcug.or g.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

[Konstantin Veretennicov]

Re: Python compilers?

"Terry Reedy" <tjreedy@udel.e du> wrote in message news:<mailman.1 83.1085238574.6 949.python-list@python.org >...[color=blue]
>
> Interesting. PyCode can be sped up by making the same assumptions.
> Richard Hettinger's recent recipe implements these assumptions. (It was
> rejected as part of the standard lib for being too implementation specific,
> but remains available in the archives and Python Cookbook site.)
>[/color]

I believe that's /Raymond/ Hettinger.

Anyway, the recipe is a great stuff:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/277940

- kv

[Jacek Generowicz]

Re: Python compilers?

Heiko Wundram <heikowu@ceosg. de> writes:
[color=blue]
> Am Dienstag, 18. Mai 2004 13:41 schrieb Jacek Generowicz:[color=green]
> > Native compilers for other languages just as dynamic as Python
> > exist. These compilers manage to achieve very significant speed
> > increases[*].[/color]
>
> In Python this isn't true. Python, instead of LISP, is "completely " dynamic,[/color]

In what way is Lisp less dynamic then Python? If what follows is the
basis of your argument, then think again?
[color=blue]
> meaning that it's pretty impossible to do type-inference for each function
> that is called (even checking types isn't possible). E.g. how do you expect
> type-inference to work with the pickle module? string -> something/Error
> would be the best description what pickle does. For the function which calls
> pickle, do you want to create versions for each possible output of Pickle?
> Which outputs of Pickle are possible?[/color]

Which outputs of the standard Common Lisp function 'read'[*] are possible?
[*] http://www.lisp.org/HyperSpec/Body/chap-23.html

[Jacek Generowicz]

Re: Python compilers?

Paul Rubin <http://phr.cx@NOSPAM.i nvalid> writes:
[color=blue]
> Carl Banks <imbosol@aerojo ckey.invalid> writes:[color=green]
> > I don't follow you. In what way is Python dynamic that Lisp isn't?[/color]
>
>[color=green][color=darkred]
> >>> class foo:[/color][/color]
> ... def bar(self, x):
> ... return x*x
> ...[color=green][color=darkred]
> >>> a = foo()
> >>> a.bar(3)[/color][/color]
> 9[color=green][color=darkred]
> >>> a.bar = lambda x: x*x*x
> >>> a.bar(3)[/color][/color]
> 27[color=green][color=darkred]
> >>>[/color][/color][/color]

I'm afraid you'll have to try harder.


[1]> (defclass foo () ())
#<STANDARD-CLASS FOO>
[2]> (defmethod bar ((foo foo) x)
(* x x))
#<STANDARD-METHOD (#<STANDARD-CLASS FOO> #<BUILT-IN-CLASS T>)>
[3]> (defparameter a (make-instance 'foo))
A
[4]> (bar a 3)
9
[5]> (defmethod bar ((self (eql a)) x)
(* x x x))
#<STANDARD-METHOD ((EQL #<FOO #x203185F9>) #<BUILT-IN-CLASS T>)>
[6]> (bar a 3)
27

[Jacek Generowicz]

Re: Python compilers?

Paul Rubin <http://phr.cx@NOSPAM.i nvalid> writes:
[color=blue]
> Carl Banks <imbosol@aerojo ckey.invalid> writes:[color=green][color=darkred]
> > > The example above kills any attempt to turn a.bar() into a static
> > > procedure call.[/color]
> >
> > Of course it does--but it's one method. A compiler, if it's good,
> > would only make the optization on methods named "bar", and it could
> > probably pare the number of possible classes it could happen to down
> > to only a few.[/color]
>
> How could it possibly know? The reassignment of a.bar could happen
> anytime, anywhere in the code. Maybe even in an eval.
>[color=green]
> > I mean you could have a Turing nightmare on your hands, with all kinds
> > of crazy setattrs and execs and stuff, in both Python and Lisp, and
> > then there's not much a compiler could do but emit totally general
> > code. I assume Lisp compilers do this sometimes.[/color]
>
> Lisp compilers might have to do that sometimes, but Python compilers
> would have to do it ALL the time. Psyco took one way out, basically
> generating code at runtime and caching it for specific operand types,
> but the result is considerable code expansion compared to precompilation.[/color]

I see how seasoned Lispers can get pretty tired of these sorts of
arguments. Please bear in mind that Lisp has been the playground for a
lot of very clever people whose aim was to solve difficult problems
efficiently ... and they've been at it for about 3 decades before
Python was even thought of. Instead of claiming that Python is
something revolutionarily new in the area of dynamicity (it isn't) and
that compiling it is impossible or futile (it isn't) have a look at
what the Lispers learned in their study of the subject over the four
decades that they have been at it. (You should only do this, of
course, if you are interested in how Python might benefit from being
compiled; if you don't care, then at least don't hurl unfounded
"opinions" about how it is impossible because Python is so amazingly
dynamic.)

Of course certain things are easier to compile to efficient machine
code than others. How does the existence of the problem spots take
away from the usefulness of compiling the easier parts? It's possible
to have significant gains. Bigger gains come with harder work. Bigger
gains can be had in some areas than in others. Certain areas are more
likely to be speed critical than others to a larger proportion of
users (number crunching, for instance). Different compiler
implementors make different choices regarding which areas of the
language they target with hard compiler optimizations.

Imagine that I would like to be able to write effecient numeric code
in pure Python, for example ... I really don't care that any instance
methods that I add will not be dispatched within a single cycle, do I?