Getting a module's byte code, how?

[Irmen de Jong]

What would be the best way, if any, to obtain
the bytecode for a given loaded module?

I can get the source:
import inspect
import os
src = inspect.getsour ce(os)

but there is no ispect.getbytec ode() ;-)

--Irmen

[Mark Nenadov]

Re: Getting a module's byte code, how?

On Wed, 02 Feb 2005 23:03:17 +0100, Irmen de Jong wrote:
[color=blue]
> What would be the best way, if any, to obtain
> the bytecode for a given loaded module?
>
> I can get the source:
> import inspect
> import os
> src = inspect.getsour ce(os)
>
> but there is no ispect.getbytec ode() ;-)
>
> --Irmen[/color]

The inspect API documentation says that code objects have "co_code", which
is a string of raw compiled bytecode.

Hope that helps!

- -
Mark Nenadov
Python Byte Solutions

http://www.pythonbyte.com/

[Irmen de Jong]

Re: Getting a module's byte code, how?

Mark Nenadov wrote:[color=blue]
> On Wed, 02 Feb 2005 23:03:17 +0100, Irmen de Jong wrote:
>
>[color=green]
>>What would be the best way, if any, to obtain
>>the bytecode for a given loaded module?
>>
>>I can get the source:
>>import inspect
>>import os
>>src = inspect.getsour ce(os)
>>
>>but there is no ispect.getbytec ode() ;-)
>>
>>--Irmen[/color]
>
>
> The inspect API documentation says that code objects have "co_code", which
> is a string of raw compiled bytecode.
>
> Hope that helps![/color]

Not very much, sorry. Because we now changed the problem into:
"how to obtain the code object from a module".


Perhaps a bit of background is in order.
For Pyro, I'm sending module byte codes across the
wire if the other side needs it (the mobile code feature).
However, this only works for modules that can be loaded
from a file on disk (because I'm now reading the .pyc file
that belongs to the module). When the receiving end in turn
calls another Pyro object, it may have to send the module's
bytecode again-- but that is no longer possible because the
module has no associated file anymore! (It is considered to
be a builtin module)
But because it is *loaded*, there must be some way to get
to the bytecodes, right?


--Irmen

[Steve Holden]

Re: Getting a module's byte code, how?

Mark Nenadov wrote:
[color=blue]
> On Wed, 02 Feb 2005 23:03:17 +0100, Irmen de Jong wrote:
>
>[color=green]
>>What would be the best way, if any, to obtain
>>the bytecode for a given loaded module?
>>
>>I can get the source:
>>import inspect
>>import os
>>src = inspect.getsour ce(os)
>>
>>but there is no ispect.getbytec ode() ;-)
>>
>>--Irmen[/color]
>
>
> The inspect API documentation says that code objects have "co_code", which
> is a string of raw compiled bytecode.
>
> Hope that helps!
>[/color]
Unfortunately co_code is an attribute of code objects, and a module
doesn't *have* a code object, as far as I know - when it's imported its
bytecode is executed in the scope of the module directory, but no
attempt is made to keep the code around thereafter: what would be the point?

Having said which, if the module was loaded from a .pyc file then the
bytecode is available from that - take everything but the first eight
bytes and use marshal.loads() to turn it back into a code object:
[color=blue][color=green][color=darkred]
>>> mbc = file("/lib/python2.4/re.pyc", "rb").read( )[8:]
>>> import marshal
>>> code = marshal.loads(m bc)
>>> code[/color][/color][/color]
<code object ? at 0xa085d60, file
"/tmp/python.2568/usr/lib/python2.4/re.py", line 1>[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

Note that the ugly details *might* change, and that byte codes are
version-dependent.

tadaaa-ly y'rs - steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.python.org/pycon/2005/
Steve Holden http://www.holdenweb.com/

[Steve Holden]

Re: Getting a module's byte code, how?

Irmen de Jong wrote:
[color=blue]
> Mark Nenadov wrote:
>[color=green]
>> On Wed, 02 Feb 2005 23:03:17 +0100, Irmen de Jong wrote:
>>
>>[color=darkred]
>>> What would be the best way, if any, to obtain
>>> the bytecode for a given loaded module?
>>>
>>> I can get the source:
>>> import inspect
>>> import os
>>> src = inspect.getsour ce(os)
>>>
>>> but there is no ispect.getbytec ode() ;-)
>>>
>>> --Irmen[/color]
>>
>>
>>
>> The inspect API documentation says that code objects have "co_code",
>> which
>> is a string of raw compiled bytecode.
>>
>> Hope that helps![/color]
>
>
> Not very much, sorry. Because we now changed the problem into:
> "how to obtain the code object from a module".
>
>
> Perhaps a bit of background is in order.
> For Pyro, I'm sending module byte codes across the
> wire if the other side needs it (the mobile code feature).
> However, this only works for modules that can be loaded
> from a file on disk (because I'm now reading the .pyc file
> that belongs to the module). When the receiving end in turn
> calls another Pyro object, it may have to send the module's
> bytecode again-- but that is no longer possible because the
> module has no associated file anymore! (It is considered to
> be a builtin module)
> But because it is *loaded*, there must be some way to get
> to the bytecodes, right?
>[/color]
I'm not sure why you think the module's code would be needed once it's
been executed. That assigns all necessary code blocks to the functions
defined therein, so the code, once the import has been executed, is
garbage (from the interpreter's rather process-centric view of things).

The module will, however, have a __file__ attribute, which should allow
you to retrieve the code as mentioned in my previous post - a .pyc file,
it appears, is just a four-byte magic number, a four-byte timestamp and
a marshalled code object.

Of course there's no guarantee that the module has been loaded from a
compiled file. In that case your only option is to read in the source
and compile it.

I am presuming that this feature of Pyro won't allow a 2.3 system to
talk to a 2.4 one, since the byte codes are incompatible.

regards
Steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.python.org/pycon/2005/
Steve Holden http://www.holdenweb.com/

[Irmen de Jong]

Re: Getting a module's byte code, how?

Steve Holden wrote:[color=blue]
> Having said which, if the module was loaded from a .pyc file then the
> bytecode is available from that - take everything but the first eight
> bytes and use marshal.loads() to turn it back into a code object:[/color]

Yup. As I explained in the other message, this is basically
what I'm doing at the moment (with a few twists; it reads the .py
file if no .pyc is available).
But I also want the bytecode of modules that don't have a .pyc file,
possibly because they have already been 'dynamically' loaded from
another bytecode string ;-)

Now, I could ofcourse store the bytecode string that I started
with *inside* the module itself, in a special attribute or so.
This just occurred to me and I think it's a possible solution.
But the bytecodes must be stored by Python itself somewhere
already... because Python is able to execute my module... right?
I want them! :-)
[color=blue]
> Note that the ugly details *might* change, and that byte codes are
> version-dependent.[/color]

I know, but this fact was not yet mentioned in the Pyro manual.
Thanks for reminding me, I'll add it.

--Irmen

[Irmen de Jong]

Re: Getting a module's byte code, how?

Steve Holden wrote:
[color=blue]
> I'm not sure why you think the module's code would be needed once it's
> been executed. That assigns all necessary code blocks to the functions
> defined therein, so the code, once the import has been executed, is
> garbage (from the interpreter's rather process-centric view of things).[/color]

Ah, ofcourse. Thanks for this insight.
I may have to rethink some of this...
[color=blue]
> I am presuming that this feature of Pyro won't allow a 2.3 system to
> talk to a 2.4 one, since the byte codes are incompatible.[/color]

Correct. However, when you're not using mobile code (the default),
you're fine.

--Irmen

[Steve Holden]

Re: Getting a module's byte code, how?

Irmen de Jong wrote:
[color=blue]
> Steve Holden wrote:
>[color=green]
>> Having said which, if the module was loaded from a .pyc file then the
>> bytecode is available from that - take everything but the first eight
>> bytes and use marshal.loads() to turn it back into a code object:[/color]
>
>
> Yup. As I explained in the other message, this is basically
> what I'm doing at the moment (with a few twists; it reads the .py
> file if no .pyc is available).
> But I also want the bytecode of modules that don't have a .pyc file,
> possibly because they have already been 'dynamically' loaded from
> another bytecode string ;-)
>[/color]
Aah, right, I suspect in these cases (which *are* pretty far from the
ordinary run of things) you'd sometimes be up the creek without a paddle.
[color=blue]
> Now, I could ofcourse store the bytecode string that I started
> with *inside* the module itself, in a special attribute or so.
> This just occurred to me and I think it's a possible solution.
> But the bytecodes must be stored by Python itself somewhere
> already... because Python is able to execute my module... right?[/color]

Not necessarily. I've just been playing with importing modules from a
database. Here's the relevant extract from my load_module() function:

code, package, path = row # ... from the database
code = marshal.loads(c ode)
module = DBmodule(modnam e) # subclass of moduleType
sys.modules[modname] = module
module.__name__ = modname
module.__file__ = path # "db:%s" % modname
module.__loader __ = dbimporter
if package:
module.__path__ = ["*db*"]
exec code in module.__dict__
#print modname, "loaded:", module, "pkg:", package
return module

Note well that the module is essentially imported by executing its
bytecode in the context of the module's directory. From that point on
the module doesn't need access to its code - all its functions and
classes have been created, and the functions and methods reachable from
the module's __dict__ now have appropriate snippets of the byte code as
their own code objects.
[color=blue]
> I want them! :-)
>[/color]
Well I'm afraid there's no guarantee that they haven't already been
garbage collected, and stamping your foot isn't going to do any good :-)
[color=blue][color=green]
>> Note that the ugly details *might* change, and that byte codes are
>> version-dependent.[/color]
>
>
> I know, but this fact was not yet mentioned in the Pyro manual.
> Thanks for reminding me, I'll add it.
>
> --Irmen[/color]

A pleasure. Thanks for Pyro! (and thanks for reminding me indirectly
that I need to guard that execution of hte module's code against
exceptions).

regards
Steve
--
Meet the Python developers and your c.l.py favorites March 23-25
Come to PyCon DC 2005 http://www.python.org/pycon/2005/
Steve Holden http://www.holdenweb.com/

[Irmen de Jong]

Re: Getting a module's byte code, how?

Steve Holden wrote:
[color=blue][color=green]
>> But I also want the bytecode of modules that don't have a .pyc file,
>> possibly because they have already been 'dynamically' loaded from
>> another bytecode string ;-)
>>[/color]
> Aah, right, I suspect in these cases (which *are* pretty far from the
> ordinary run of things) you'd sometimes be up the creek without a paddle.[/color]

I was afraid of that.
The whole mobile code stuff in Pyro is very powerful, for the
cases where it works and makes sense, I think.
However it is also fairly messy.
In the end perhaps it introduces more problems than it solves,
but its really nice to see it work for the simple cases :)

[color=blue][color=green]
>> I want them! :-)
>>[/color]
> Well I'm afraid there's no guarantee that they haven't already been
> garbage collected, and stamping your foot isn't going to do any good :-)[/color]

Heh, that was funny :D
[color=blue]
> A pleasure. Thanks for Pyro! (and thanks for reminding me indirectly
> that I need to guard that execution of hte module's code against
> exceptions).[/color]

Pyro happily throws these exceptions in your face.
Like I said, rather messy.

--Irmen