Python bytecode compatibility between interpreter versions

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Andrew MacIntyre wrote:

[color=blue]
> - Python has always been specified in terms of the language, with a
> portable reference interpreter implementation for that language (which
> just happens to compile to bytecode, and save that bytecode to disk
> as an optimisation).
>
> IOW, the bytecode is an implementation detail for _one_ interpreter
> implementation.
>
> This becomes obvious when you consider the possibilities associated
> with Jython, which could (already does?) compile Python source to jar
> files directly.[/color]

Contrary to what you seem to be implying here, Guido never expressly designed
Python to be easily portable to run on top of other bytecode implementations .
Jython was a very difficult feat to pull and the same goes (probably moreso)
for Python.NET.

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Roger Binns wrote:
[color=blue]
> You are considering Python .pyc files and Java class files to be
> the same thing. They aren't.[/color]

They're not 'the same thing'. They are roughly analogous to one
another though, wouldn't you say?
[color=blue]
> pyc files are a behind the scenes
> implementation detail of one version of the Python interpreter
> that are automatically generated and maintained.[/color]

Huh?? .pyc files are an "implementa tion detail of the Python
interpreter"? Care to expound further? 'Coz that sounds
pretty nonsensical...

[Skip Montanaro]

Re: Python bytecode compatibility between interpreter versions


Jon> [ expand jar file - presto! instant app ]

Aahz> .... I am sure that simple cases work as you describe, but I'd
Aahz> like a cite for Java working as you describe in general.

Jon> Yes. Java works like that ... [ mentions Batik ]

The fundamental difference between Java and Python in this regard is that
the JVM must absolutely be part of the defined API since it is delivered
separate from the compiler. Whoever does this stuff probably has to go to
great lengths to preserve compatibility. This is not the case for the
Python VM, since a new version is delivered with each distribution.

Skip

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Skip Montanaro wrote:
[color=blue]
> The fundamental difference between Java and Python in this regard is that
> the JVM must absolutely be part of the defined API since it is delivered
> separate from the compiler.[/color]

Absolutely untrue. The JVM is part of the Java /spec/, but has nothing to do
with the /API/ (e.g. Java 2 Standard Edition) that application programmers use.
[color=blue]
> Whoever does this stuff probably has to go to great lengths to preserve
> compatibility. This is not the case for the Python VM, since a new version
> is delivered with each distribution.[/color]

I don't get what you're trying to say here. The JVM has changed very little
(or not at all) over the different Java versions. Neither the incremental
feature-additions to the Java compiler nor modifications to the Java platform
APIs require any great effort to 'maintain compatibility' between them and the
JVM. Even fairly major additions like generics (i.e. Pizza compiler) do not
sound like they require ripping out great chunks of guts and replacing them with
something totally new.

If anything though, what you're saying might apply more to Jython. New Python
language features tend to be more radical / come faster than new Java features
but the Jython authors will have to implement these on a JVM that cannot morph
to accomodate such new features. Still, Jython's continued development proves
that it is possible implement Python's features on a fixed underlying VM.

The only reason I can think of for wanting CPython's VM to remain undocumented /
not-nailed-down is because we want to make it easier / give more freedom to the
language designers to implement the plumbing for new Python features.

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Terry Reedy wrote:
[color=blue]
> The CPython VM is intentionally underdocumented and intentionally not part
> of the language spec. It is still being developed and might radically
> change in the future. There will probably be minor changes in the next
> release as a result of experiments designed to find combinations of codes
> that dependably run faster on all major platforms. However, the current
> byte code is described in the Library Ref as part of the doc for the dis
> (assembly) module.[/color]

This is the answer I was looking for... :-) Thanks, Terry.

[Skip Montanaro]

Re: Python bytecode compatibility between interpreter versions

[color=blue][color=green]
>> pyc files are a behind the scenes implementation detail of one
>> version of the Python interpreter that are automatically generated
>> and maintained.[/color][/color]

Jon> Huh?? .pyc files are an "implementa tion detail of the Python
Jon> interpreter"? Care to expound further? 'Coz that sounds pretty
Jon> nonsensical...

As I mentioned in my previous post, the distiction between the Java VM and
the Python VM is that the former is delivered separate from the compiler and
other tools, while the latter is delivered afresh with each Python
distribution. Guido would be free to eliminate the Python VM from the next
version of Python and instead directly interpret the abstract syntax tree
emitted by the compiler. PyChecker and other tools which operate directly
with the bytecode would break, but Python applications should still work.

Think of it this way. The Java virtual machine architecture is sort of akin
to the current Wintel architecture. The Python virtual machine architecture
would like Microsoft shipping a new, possibly incompatible, CPU with each
copy of Windows.

Skip

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Andrew MacIntyre wrote:
[color=blue]
> This becomes obvious when you consider the possibilities associated
> with Jython, which could (already does?) compile Python source to jar
> files directly.[/color]

A .jar file is just a .zip file composed of a collection of .class files
(JVM bytecodes) plus an additional text-based manifest file.

Jython _does_ compile Python source to .class files (because that is
really the only way to get it to run on the JVM), and the same will apply
for any other language, like Kawa (a Scheme implementation) , that run 'on
top of Java' (i.e. gain access to the huge functionality present in
the Java libraries).

Essentially, whenever someone says so-and-so language runs on Java, it means
that they have a compiler for that langauge that emits JVM bytecodes. These
bytecodes can then be run on any implementation of the JVM, whether a JIT-compiling
one or purely interpreted. Further, you can also opt to take that bytecode / .class
file and compile it into native code via a native-code compiler like gcj.

The separation of the layers is very nice and clean and you can substitute
those coming from different sources and expect things to work:


Language Compiler (javac, Jikes, Pizza, Jython, Kawa, JIcon, etc...)
|
| emits .class files which can be packaged into .jars
|
V
JVM implementation + Runtime Environment / native code compiler
(Hotspot, Jalapeno, etc...) (gcj, JRockit, etc...)
|
|
V
native executable

Thus it is reasonable to take the exact same .class file compiled by, say,
Jython or Kawa on a Mac and expect it to run perfectly well on a JVM
implementation running on an x86 or an AS/400 minicomputer. *This*
is what would be nice to have with .pyc files.

Note that there is one other consideration besides the presence of a JVM,
and that is the Runtime Environment (the APIs). Swing may not have been
implemented on the AS/400, but it is implemented on X Window so the
Jython-Swing app you compiled on a Mac would work and look exactly alike
( sluggishly, that is... ;-) ) on a Solaris, FreeBSD, or Linux box.

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Skip Montanaro wrote:
[color=blue]
> Jon> [ expand jar file - presto! instant app ][/color]

Also to clarify, describing

java -jar jarfile

as expanding a jarfile is not quite right. It works
exactly as you would run an executable. On many
platforms, you can even just type:

jarfile.jar

on the command line or double click a jar icon to start the
app(*). Not once do you have to worry about extra files
lying around. In fact, it might even be the case that the
class files are loaded into the jvm straight from the
jarfile and not decompressed behind the scenes into a temporary
directory at all.


(*) That's why I think Web Start is a stupid idea that
makes people think Java is more complicated than it really
is.

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Roger Binns wrote:

[color=blue]
> Modulo other dependencies also being installed as well.
>
> You can do the same thing with Python. Just package up the source
> files as a zip.
>
> $ PYTHONPATH=exam ple.zip python main.py[/color]

Just like Java's CLASSPATH (and possibly inspired by it)... this is a
great tip. Does it work on Windows as well? And how about .tar.gz,
..tar.bz or .gzip files?

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Skip Montanaro wrote:
[color=blue]
> As I mentioned in my previous post, the distiction between the Java VM and
> the Python VM is that the former is delivered separate from the compiler and
> other tools, while the latter is delivered afresh with each Python
> distribution.[/color]

Actually, Sun always bundles Hotspot and javac together in an SDK release.
But yes, you _can_ get and use a compiler separately (Jikes, for example)
from the JVM and there is very little version dependency between the two
(the JVM has stayed the same for a long time now).

[color=blue]
> Guido would be free to eliminate the Python VM from the next
> version of Python and instead directly interpret the abstract syntax tree
> emitted by the compiler.[/color]

eh...? If interpreting the AST isn't what the CPython VM does, then what
does it do?

[color=blue]
> PyChecker and other tools which operate directly with the bytecode would
> break, but Python applications should still work.[/color]

Does this mean that you will need a different version of PyChecker for each
Python .x release?

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Skip Montanaro wrote:

Jon> Huh?? .pyc files are an "implementa tion detail of the Python
Jon> interpreter"? Care to expound further? 'Coz that sounds pretty
Jon> nonsensical...

"The internal structure of a .pyc file depends on the implementation
details of the particular version of the Python interpreter that it was
compiled with"

now when you phrase it that way, it makes sense...

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Anyway, we can at least count on the fact that a .pyc compiled
by a particular Python version (say, Python 2.1.1) on one platform
(say, Mac), will run on another platform (say, Solaris) as long
as the interpreter has the same minor version (say, Python 2.1.0).

uh... right?

[Skip Montanaro]

Re: Python bytecode compatibility between interpreter versions


Jon> Anyway, we can at least count on the fact that a .pyc compiled by a
Jon> particular Python version (say, Python 2.1.1) on one platform (say,
Jon> Mac), will run on another platform (say, Solaris) as long as the
Jon> interpreter has the same minor version (say, Python 2.1.0).

Yup. I believe there were some problems between 32- and 64-bit platforms
ages ago, but they've been resolved.

Skip

[Skip Montanaro]

Re: Python bytecode compatibility between interpreter versions

[color=blue][color=green]
>> The fundamental difference between Java and Python in this regard is
>> that the JVM must absolutely be part of the defined API since it is
>> delivered separate from the compiler.[/color][/color]

Jon> Absolutely untrue. The JVM is part of the Java /spec/, but has
Jon> nothing to do with the /API/ (e.g. Java 2 Standard Edition) that
Jon> application programmers use.

Sorry, terminology flub. Spec, external interface, API... What I meant was
that the Java folks obligated themselves to keep it compatible. There are
no such claims for the Python VM.
[color=blue][color=green]
>> Whoever does this stuff probably has to go to great lengths to
>> preserve compatibility. This is not the case for the Python VM,
>> since a new version is delivered with each distribution.[/color][/color]

Jon> I don't get what you're trying to say here. The JVM has changed
Jon> very little (or not at all) over the different Java versions.

The Python VM changes frequently, certainly when compared to the Java VM.
The PVM is an implementation detail. Your experience with Batik could not
in general be replicated with a tar file full of Python bytecode, and I'm
not sure I'd want it to be.

Jon> The only reason I can think of for wanting CPython's VM to remain
Jon> undocumented / not-nailed-down is because we want to make it easier
Jon> / give more freedom to the language designers to implement the
Jon> plumbing for new Python features.

Exactly.

Skip

[Skip Montanaro]

Re: Python bytecode compatibility between interpreter versions

[color=blue][color=green]
>> Guido would be free to eliminate the Python VM from the next version
>> of Python and instead directly interpret the abstract syntax tree
>> emitted by the compiler.[/color][/color]

Jon> eh...? If interpreting the AST isn't what the CPython VM does,
Jon> then what does it do?

It interprets the Python bytecodes.
[color=blue][color=green]
>> PyChecker and other tools which operate directly with the bytecode
>> would break, but Python applications should still work.[/color][/color]

Jon> Does this mean that you will need a different version of PyChecker
Jon> for each Python .x release?

Not necessarily. It doesn't know everything there is to know about Python
bytecode, but it does know some bytecodes and rummages around in code
objects a bit, so if there was no bytecode it would have nothing to rummage
through.

Skip