Python bytecode compatibility between interpreter versions

[Roger Binns]

Re: Python bytecode compatibility between interpreter versions

> > You are considering Python .pyc files and Java class files to be[color=blue][color=green]
> > the same thing. They aren't.[/color]
>
> They're not 'the same thing'. They are roughly analogous to one
> another though, wouldn't you say?[/color]

No. I explained it further in the message.
[color=blue]
> Huh?? .pyc files are an "implementa tion detail of the Python
> interpreter"? Care to expound further? 'Coz that sounds
> pretty nonsensical...[/color]

One version of a compliant Python interpretter happens to use
these files, and expose that they exist. They are generated on
demand and automatically updated. The Python interpretter could
have stored them elsewhere on the filesystem (for example in
a hidden "cache" directory), but for simplicity's sake stored
them alongside the source file.

Go ahead and write protect the directories containing your
Python source (after deleting all .pyc files). You will find
everything runs just fine without any visible .pyc files.

If you write your own Python interpretter/compiler, it will be
considered compliant if it implements the language correctly.
There is *nothing* about bytecodes, or bytecode compatibility.
For example if you wanted to write your own that ran on top of
Java, you would probably just directly generate class files.
If you wrote one that ran on a Palm device you would do something
else. It is up to your implementation what it does. The
only requirement would be working with Python *source* files.

Roger

[Roger Binns]

Re: Python bytecode compatibility between interpreter versions

> > $ PYTHONPATH=exam ple.zip python main.py[color=blue]
>
> Just like Java's CLASSPATH (and possibly inspired by it)...[/color]

Except predating Java by several years ...
[color=blue]
> Does it work on Windows as well?[/color]

C:\WINDOWS> set PYTHONPATH=exam ple.zip
C:\WINDOWS> python main.py
[color=blue]
> And how about .tar.gz, .tar.bz or .gzip files?[/color]

Nope. The unit of transport is zip files. You can also
point to expanded directories.

Roger

[Roger Binns]

Re: Python bytecode compatibility between interpreter versions

Jon Perez wrote:[color=blue]
> Anyway, we can at least count on the fact that a .pyc compiled
> by a particular Python version (say, Python 2.1.1) on one platform
> (say, Mac), will run on another platform (say, Solaris) as long
> as the interpreter has the same minor version (say, Python 2.1.0).
>
> uh... right?[/color]

Nope. They are an implementation specific thing. The interpretter
puts version and other information in them, and regenerates them
on the fly. There is nothing wrong with an interpretter deciding
to regenerate them every single day in a different format, or
based on the moon phase, or your login id or anything else.

Roger

[Roger Binns]

Re: Python bytecode compatibility between interpreter versions

> Jon> Huh?? .pyc files are an "implementa tion detail of the Python[color=blue]
> Jon> interpreter"? Care to expound further? 'Coz that sounds pretty
> Jon> nonsensical...
>
> "The internal structure of a .pyc file depends on the implementation
> details of the particular version of the Python interpreter that it was
> compiled with"
>
> now when you phrase it that way, it makes sense...[/color]

No that is not what I meant. What I meant is that the very existence
of .pyc files is an implementation detail of one version of the Python
interpretter. Tomorrow it could use 3 different files, or none at all.
They are pretty much an optional behind the scenes performance optimisation
for CPython. There is no requirement for them to exist.

Tomorrow I could write my own Python interpretter that generates
..tok files, .optim files and .silly files. As long as it executes
Python *source* (ie .py files) then those are an implementation
detail.

In Java, the JVM only executes bytecode. In CPython it only
executes .py files. Behind the scenes as an implementation
optimisation, the .py files are tokenised and transformed into
bytecode and used as internal representation. As an implementation
specific performance measure, they are written out to .pyc
files which are loaded if not out of date in the future instead
of re-parsing the .py files.

You could delete every .pyc file on your system, and then mark
your disk as read-only and your Python programs will run fine.

Java does something similar behind the scenes with HotSpot.
It compiles things to native code, and keeps profiling information.
They are not written out to disk and hence are not persistent
between executions, but if they were the usage would be analogous
to .pyc files.

Roger

[Joe Mason]

Re: Python bytecode compatibility between interpreter versions

In article <c3k8m2$28mtia$ 1@ID-180326.news.uni-berlin.de>, Jon Perez wrote:[color=blue][color=green]
>> Really? I'm sorry, I have just a tiny bit of difficulty believing that,
>> considering all the stories i've heard about Java not being completely
>> cross-platform and cross-version. I am sure that simple cases work as
>> you describe, but I'd like a cite for Java working as you describe in
>> general.[/color]
>
> Yes. Java works like that and has for a long time already. One of the
> most notable examples i can give is the SVG viewer Batik (not a trivial
> program). Worked liked a charm on both Linux and Win32, and it does not
> rely on an exact JRE on which it will run on properly. Virtually all
> the Swing apps you can try out will work just as reliably and identically
> on different platforms as Batik.[/color]

This has not been remotely my experience. Recently I tried 5 different
Linux JVM's for one program, leaving the Sun JRE until last because it
was a bit harder to install, and only the Sun one worked.

When a program *does* work right on multiple JVM's (Limewire, for
instance), I attribute it to the skill of the packagers. The Limewire
team obviously put a lot of effort into that, just by looking at the
number of downloads and installers on their page.

Joe

[Joe Mason]

Re: Python bytecode compatibility between interpreter versions

In article <slrnc5sc8c.qbu .joe@gate.notch arles.ca>, Joe Mason wrote:[color=blue]
> This has not been remotely my experience. Recently I tried 5 different
> Linux JVM's for one program, leaving the Sun JRE until last because it
> was a bit harder to install, and only the Sun one worked.[/color]

(I meant to add the name of the program in question, but I can't find it
now.)

Joe

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Roger Binns wrote:
[color=blue][color=green]
>>"The internal structure of a .pyc file depends on the implementation
>>details of the particular version of the Python interpreter that it was
>>compiled with"
>>
>>now when you phrase it that way, it makes sense...[/color]
>
> No that is not what I meant. What I meant is that the very existence
> of .pyc files is an implementation detail of one version of the Python
> interpretter. Tomorrow it could use 3 different files, or none at all.
> They are pretty much an optional behind the scenes performance optimisation
> for CPython. There is no requirement for them to exist.[/color]

Is it really the case that it isn't even recommended to distribute one's
application as .pyc files?

In commercial apps for which one might not want to distribute the source
for, it would be really nice if one can count on just requiring the end-user
to install a version of Python (but not extra extension modules anymore) and
then distribute the app as a bunch of .pyc's packaged in a .zip file and
use the PYTHONPATH method you showed. Just remember to provide a set of
..pyc's for each version of Python you support.
[color=blue]
> Java does something similar behind the scenes with HotSpot.
> It compiles things to native code, and keeps profiling information.
> They are not written out to disk and hence are not persistent
> between executions, but if they were the usage would be analogous
> to .pyc files.[/color]

In Java, profiling info may not be written out to disk, but the
bytecodes certainly are (they are what is in the class files). Which
is roughly equivalent to what happens with .pyc files, correct? .pyc
files don't contain profiling info, nor was I ever aware that the Python
VM even does run-time behaviour profiling... (Pysco may, though)
[color=blue]
> If you write your own Python interpretter/compiler, it will be
> considered compliant if it implements the language correctly.
> There is *nothing* about bytecodes, or bytecode compatibility.
> For example if you wanted to write your own that ran on top of
> Java, you would probably just directly generate class files.
> If you wrote one that ran on a Palm device you would do something
> else. It is up to your implementation what it does. The
> only requirement would be working with Python *source* files.[/color]

Okay, I get it. "Nothing in the spec specifies that .pyc files
have to exist". Further, "even the existence of a VM representation
is not part of the Python spec".

In practice though, there is really just one implementation of Python
(with the standard libraries... excluding Jython) which works using the
same bytecode mechanism on _all_ platforms, so as far as I can tell,
it would be quite feasible to distribute applications in .pyc form.

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Joe Mason wrote:[color=blue]
> In article <slrnc5sc8c.qbu .joe@gate.notch arles.ca>, Joe Mason wrote:
>[color=green]
>>This has not been remotely my experience. Recently I tried 5 different
>>Linux JVM's for one program, leaving the Sun JRE until last because it
>>was a bit harder to install, and only the Sun one worked.[/color][/color]

If you stick to console programs, you are not likely to experience
problems. However, when it comes to Swing apps, the other JREs
(JVM + J2SE API implementation) out there (with the possible
exception of IBM's) are not mature enough to be practical in use.

The bottom line is that by sticking to Sun's JRE on Linux and Windows,
and Apple's implementation on the OS X, you do have a very effective
WORA platform. The others are largely good only for experimentation .
[color=blue][color=green]
>> the Sun JRE until last because it
>> was a bit harder to install, and only the Sun one worked.[/color][/color]

Strange, I didn't find the Sun JRE the least bit difficult
to install on Linux... execute the script, add the j2sdk/ or
j2re/ and/or its bin directory to your PATH, and you can start
doing

java -jar jarfile

under X Window, and run Swing apps immediately.

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Roger Binns wrote:
[color=blue]
> Nope. They are an implementation specific thing. The interpretter
> puts version and other information in them, and regenerates them
> on the fly.
>
> There is nothing wrong with an interpretter deciding
> to regenerate them every single day in a different format, or
> based on the moon phase, or your login id or anything else.[/color]

So we can't even count on the fact that Python 2.2 running under Windows
on an Athlon will generate a .pyc file that's identical with that
generated by the exact same version (2.2) of the interpreter running
under Linux on, say, a Pentium III, and further, is not even *guaranteed*
to run on it?!?

This does not seem to fit in with what I've been reading... it would
be a lot more logical (and convenient) if we can count on .pyc portability
across different _patch versions_ (.x.y) of the same minor version (.x)
of CPython on whatever platform it is running on, and I believe that
Python's contributors would make an effort to ensure that this is the case.

[Roger Binns]

Re: Python bytecode compatibility between interpreter versions

> Is it really the case that it isn't even recommended to distribute one's[color=blue]
> application as .pyc files?
>
> In commercial apps for which one might not want to distribute the source
> for, it would be really nice if one can count on just requiring the end-user
> to install a version of Python (but not extra extension modules anymore) and
> then distribute the app as a bunch of .pyc's packaged in a .zip file and
> use the PYTHONPATH method you showed. Just remember to provide a set of
> .pyc's for each version of Python you support.[/color]

In the Java world, the JVM is a seperate installable component.
Some installers like ZeroG will install them for you. In general
all Java bytecode is expected (but not required) to share the
main system JVM.

In Python, if you want to distribute an application, and you
don't want the user to have to install Python or any extensions,
you use one of the installer generators out there. They gather
up the interpretter, your code and extension libraries and dump
them all into one directory. They then include a stub that makes
them all work together. It especially has no reliance on any
pre-installed interpretter.

If you want to see this in action, try my project at
http://bitpim.sf.net The downloads give you a program that runs
on Windows, Mac and Linux without any prerequisites. If you
examine the install directory, you will find the find Python
interpretter DLL or binary, all needed extension shared libraries
and the program code (often as .pyc files in a zip archive or
appended to the main binary).

Note that the installer generator happens to know what formats
and other quirks the Python interpretter works with which is why
it does include some .pyc's.
[color=blue]
> In Java, profiling info may not be written out to disk, but the
> bytecodes certainly are (they are what is in the class files). Which
> is roughly equivalent to what happens with .pyc files, correct?[/color]

You are pedantically correct in that they both contain bytecode, but
the level is different. *If* the JVM wrote out HotSpot compilations
and profiling information, the resulting files would be analogous to
..pyc files. They would not be *required* for execution, and they
would be automatically regenerated if necessary (eg if you changed
source files). In theory it could then execute from them without
requiring the original class files, but that would be an implementation
specific thing.
[color=blue]
> .pyc
> files don't contain profiling info, nor was I ever aware that the Python
> VM even does run-time behaviour profiling... (Pysco may, though)[/color]

I don't know of any implementations that do, but the implementations
are free to do so. As I keep saying, .pyc files are an implementation
specific detail of one version of one implementation of the Python
interpretter. It can make as many intermediate file formats and contents
as it wants, as can any other implementation. The only requirement for
a conforming Python implementation is that it execute Python *source*,
and how it does it under the scenes is not specified.
[color=blue]
> In practice though, there is really just one implementation of Python
> (with the standard libraries... excluding Jython) which works using the
> same bytecode mechanism on _all_ platforms, so as far as I can tell,
> it would be quite feasible to distribute applications in .pyc form.[/color]

If you distribute the exact interpretter that generated those exact
..pyc files then you will be fine in practise. That is what the various
installer generators out there do. Some examples are py2exe, cx_Freeze
and BundleBuilder which is what my project uses on Windows, Linux and
Mac respectively.

With the addition of zip imports in Python 2.3, I think things will
lead to a unification of the installer generators out there (there have
been several different ones, often limited to a subset of platforms for
historical reasons).

Roger

[Roger Binns]

Re: Python bytecode compatibility between interpreter versions

> So we can't even count on the fact that Python 2.2 running under Windows[color=blue]
> on an Athlon will generate a .pyc file that's identical with that
> generated by the exact same version (2.2) of the interpreter running
> under Linux on, say, a Pentium III, and further, is not even *guaranteed*
> to run on it?!?[/color]

There are no guarantees that the same .pyc files will work the next day
with the same interpretter. The interpretter could change the format
every single day if the implementor so wanted.
[color=blue]
> This does not seem to fit in with what I've been reading... it would
> be a lot more logical (and convenient) if we can count on .pyc portability
> across different _patch versions_ (.x.y) of the same minor version (.x)
> of CPython on whatever platform it is running on,[/color]

Now in practise the .pyc files are not changed every day because there
is no need, but an implementation is free to do so. In many cases they
are the same between platforms and minor versions, but that is not
a specification or a requirement.
[color=blue]
> and I believe that
> Python's contributors would make an effort to ensure that this is the case.[/color]

Why? To distribute executable Python code, you distribute the .py files.
To distribute executable Java code, you distribute .class files. They
can be wrapped up in .zip and .jar archives respectively.

Java is defined as a compiled language. Python is not a compiled
language, although an implementation is free to do so. If you really
really want a compiled form of Python, try Jython which compiles to
Java .class files.

Roger

[Ronald Oussoren]

Re: Python bytecode compatibility between interpreter versions

>> In practice though, there is really just one implementation of Python[color=blue][color=green]
>> (with the standard libraries... excluding Jython)[/color][/color]

That's *two* implementations of Python: CPython and Jython, one of
which uses pyc files. You're conveniently ignoring the primary example
of a Python interpreter that does not use pyc files.
[color=blue][color=green]
>> which works using the
>> same bytecode mechanism on _all_ platforms, so as far as I can tell,
>> it would be quite feasible to distribute applications in .pyc form.[/color][/color]

Yes it is feasible to distribute apps in .pyc form, as long as you
target a specific version of Python.

Ronald

[Terry Reedy]

Re: Python bytecode compatibility between interpreter versions


"Ronald Oussoren" <oussoren@cistr on.nl> wrote in message
news:DD2E4369-7BE3-11D8-8375-0003931CFE24@ci stron.nl...[color=blue][color=green][color=darkred]
> >> In practice though, there is really just one implementation of Python
> >> (with the standard libraries... excluding Jython)[/color][/color][/color]

People who make their living with Jython would find the exclusion puzzling
;-)
[color=blue]
> That's *two* implementations of Python: CPython and Jython, one of
> which uses pyc files. You're conveniently ignoring the primary example
> of a Python interpreter that does not use pyc files.[/color]

There are actually two more:

3) Viper (now dead, I believe), which translated Python (+ 'enhancements')
to, I believe, with some fuzzyness, OCaml.

4) PyPy (as yet only partial, I believe), which, if funded for development
and spectacularly successful, could become the reference implementation
(but that is at best years away).

Nonetheless, CPython is now the main, reference implementation. 2.x.y .pyc
files are cross-platform portable. I believe the same is also true
regardless of the '.y' part. Reading PyDev, I have seen no evidence that
the planned 2.4 will be very much different.

Terry J. Reedy

[Jon Perez]

Re: Python bytecode compatibility between interpreter versions

Roger Binns wrote:
[color=blue]
> In the Java world, the JVM is a seperate installable component.
> Some installers like ZeroG will install them for you. In general
> all Java bytecode is expected (but not required) to share the
> main system JVM.[/color]

Afaik, each app will start its own copy of a JVM.
[color=blue]
> In Python, if you want to distribute an application, and you
> don't want the user to have to install Python or any extensions,
> you use one of the installer generators out there. They gather
> up the interpretter, your code and extension libraries and dump
> them all into one directory. They then include a stub that makes
> them all work together. It especially has no reliance on any
> pre-installed interpretter.[/color]

Like I mentioned earlier, I am aware of 'freezing' but having to bundle
an interpreter with every app is exactly what I would rather avoid...
[color=blue]
> If you want to see this in action, try my project at
> http://bitpim.sf.net The downloads give you a program that runs
> on Windows, Mac and Linux without any prerequisites.[/color]

What would be great is to not have to choose between a Windows,
Mac or Linux or XXX download (like in a proper Java app).
[color=blue]
> You are pedantically correct in that they both contain bytecode, but
> the level is different. *If* the JVM wrote out HotSpot compilations
> and profiling information, the resulting files would be analogous to
> .pyc files.[/color]

But you're not saying that .pyc files contain profiling information,
right? Because that is what it's sounding like...
[color=blue]
> They would not be *required* for execution, and they
> would be automatically regenerated if necessary (eg if you changed
> source files). In theory it could then execute from them without
> requiring the original class files, but that would be an implementation
> specific thing.[/color]

Yes, this is the main difference between Java and Python. Python
prefers the source to be present, Java doesn't care.
[color=blue]
> With the addition of zip imports in Python 2.3,[/color]

Correct me if I'm wrong, but I thought you just said earlier that
zip imports predated Java by several years and now you're saying
they've just been added to Python 2.3?

[Terry Reedy]

Re: Python bytecode compatibility between interpreter versions


"Jon Perez" <jbperez808@yah oo.com> wrote in message
news:c3lguo$271 vv8$1@ID-180326.news.uni-berlin.de...[color=blue]
> In commercial apps for which one might not want to distribute the source
> for, it would be really nice if one can count on just requiring the[/color]
end-user[color=blue]
> to install a version of Python (but not extra extension modules anymore)[/color]
and[color=blue]
> then distribute the app as a bunch of .pyc's packaged in a .zip file and
> use the PYTHONPATH method you showed. Just remember to provide a set of
> .pyc's for each version of Python you support.[/color]

The issue of hiding source comes up every couple of months. If there is
not a FAQ entry, there are certainly many past threads. Note that the dis
modules makes bytecode pretty easy to read, especially with some
experience.

Terry J. Reedy