Getting better traceback info on exec and execfile - introspection?

[R. Bernstein]

In doing the extension to the python debugger which I have here:

http://sourceforge.net/project/showfiles.php?group_id=61395&package_id=175827

I came across one little thing that it would be nice to get done better.

I notice on stack traces and tracebacks, an exec or execfile command
appears as a stack entry -- probably as it should since it creates a
new environment.

However the frame information for exec or execfile looks like this:
File "<string>", line 1, in ?

Okay, the "line 1" probably changes, but most of the time it's going
to be one. And in code that's in CVS (cvsview:
http://cvs.sourceforge.net/viewcvs.py/bashdb/pydb/) you'll see that
instead of this I now put some thing like:

## 1 in exec '<string>' at line 1

which is perhaps is a little more honest since one is not really in a
file called <string>. However the way the debugger gets this *is*
still a little hoaky in that it looks for something in the frame's
f_code.co_filen ame *called* <string>. And from that it *assumes* this
is an exec, so it can't really tell the difference between an exec
command an execfile command, or a file called <string>. But I suppose
a little more hoakiness *could* be added to look at the outer frame
and parse the source line for "exec" or "execfile".

And suppose instead of '<string>' I'd like to give the value or the
leading prefix of the value instead of the unhelpful word '<string>'?
How would one do that? Again, one way is to go into the outer frame
get the source line (if that exists), parse that and interpolate
argument to exec(file). Is there are better way?

Another and perhaps more direct way to distinguish exec/execfile might
be to some how look at the byte code of the frame entry and decode
that. It also could be used for example in another place in the
debugger where one wants to skip over executing "def" statements (not
the function invocation, but the statement which adds the
function/method so it can get called). But depending on how things are
done unless there are nice version-independent and
implementation-independent symbolic definitions for exec_opcode,
execfile_opcode and def_opcode, although straight-forward and reliable
for a given version/implementation it might not be a good thing to do.

Any help, thoughts or pointers?

[R. Bernstein]

Re: Getting better traceback info on exec and execfile - introspection?

I suggested:[color=blue]
> And suppose instead of '<string>' I'd like to give the value or the
> leading prefix of the value instead of the unhelpful word '<string>'?
> How would one do that? Again, one way is to go into the outer frame
> get the source line (if that exists), parse that and interpolate
> argument to exec(file). Is there are better way?[/color]

I've hacked this approach into the CVS for pydb, but it is not all
that accurate (although in practice I'm pleased with the result).

My current regular expression is (^|\s+)exec\s+( .*) which does match
simple things, but it would fail on say by *not* matching:
x=1;exec "y=2";exec "z=3"
and fail by *wrongly* matching on:
x=" exec meeting in 5 mins"

Even if I knew how to call the python parser on the line, that still
might not be good enough in the presence of continuation lines.

This probably should be addressed at a lower level in gathering
traceback data inside Python.
[color=blue]
> Another and perhaps more direct way to distinguish exec/execfile might[/color]
^^^^^^^^
execfile is not a problem, just exec.

[Fernando Perez]

Re: Getting better traceback info on exec and execfile -introspection?

R. Bernstein wrote:
[color=blue]
> In doing the extension to the python debugger which I have here:
>[/color]
http://sourceforge.net/project/showf...kage_id=175827[color=blue]
> I came across one little thing that it would be nice to get done better.
>
> I notice on stack traces and tracebacks, an exec or execfile command
> appears as a stack entry -- probably as it should since it creates a
> new environment.
>
> However the frame information for exec or execfile looks like this:
> File "<string>", line 1, in ?[/color]

That comes from how the code object was compiled:
[color=blue][color=green][color=darkred]
>>> code = compile('error' ,'Anything you want goes here','single')
>>> exec code[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "Anything you want goes here", line 1, in ?
NameError: name 'error' is not defined


For example, in ipython:

In [1]: error
---------------------------------------------------------------------------
exceptions.Name Error Traceback (most recent
call last)

/home/fperez/<ipython console>

NameError: name 'error' is not defined


So any regexp-matching based approach here is likely to be fairly brittle,
unless you restrict your tool to the standard python interpreter, and you
get some guarantee that it will always tag interactive code with
'<string>'.

Regards,

f

[R. Bernstein]

Re: Getting better traceback info on exec and execfile - introspection?

Fernando Perez <fperez.net@gma il.com> writes:[color=blue]
> R. Bernstein wrote:[/color]
....[color=blue][color=green]
> > However the frame information for exec or execfile looks like this:
> > File "<string>", line 1, in ?[/color]
>
> That comes from how the code object was compiled:[/color]
....[color=blue]
> So any regexp-matching based approach here is likely to be fairly brittle,
> unless you restrict your tool to the standard python interpreter, and you
> get some guarantee that it will always tag interactive code with
> '<string>'.[/color]

Thanks for the information! Alas, that's bad news. The question then
remains as to how to accurately determine if a frame is running an
exec operation and accurately get the string associated with the exec.

Given all the introspection about python's introspection, it should be
possible.

I also tried pydb.py using IPython 0.7.0 (using an automatically
updated Fedora Core 5 RPM) and although I was expecting pydb.py breakage
from what you report, I didn't find any difference.


% ipython
Python 2.4.2 (#1, Dec 17 2005, 13:02:22)
Type "copyright" , "credits" or "license" for more information.

IPython 0.7.0 -- An enhanced Interactive Python.
? -> Introduction to IPython's features.
%magic -> Information about IPython's 'magic' % functions.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]: %run /usr/lib/python2.4/site-packages/pydb.py test1.py[color=blue]
> /home/rocky/python/test1.py(2)?()[/color]
-> import sys
(Pydb) where
-> 0 in file '/home/rocky/python/test1.py' at line 2
## 1 in exec cmd in globals, locals at line 1
## 2 run() called from file '/usr/lib/python2.4/bdb.py' at line 366
(Pydb)

Note entry ##1 is *not*
File "<string>", line 1, in ?

So somehow I guess ipython is not intercepting or changing how compile
was run here.

[Fernando Perez]

Re: Getting better traceback info on exec and execfile -introspection?

R. Bernstein wrote:
[color=blue]
> Fernando Perez <fperez.net@gma il.com> writes:[color=green]
>> R. Bernstein wrote:[/color]
> ...[color=green][color=darkred]
>> > However the frame information for exec or execfile looks like this:
>> > File "<string>", line 1, in ?[/color]
>>
>> That comes from how the code object was compiled:[/color]
> ...[color=green]
>> So any regexp-matching based approach here is likely to be fairly
>> brittle, unless you restrict your tool to the standard python
>> interpreter, and you get some guarantee that it will always tag
>> interactive code with '<string>'.[/color]
>
> Thanks for the information! Alas, that's bad news. The question then
> remains as to how to accurately determine if a frame is running an
> exec operation and accurately get the string associated with the exec.
>
> Given all the introspection about python's introspection, it should be
> possible.[/color]

I thought a little about this. One possibility would be to check whether
the argument given in the string exists as a file; if not, assume it's a
string. I can't think of another way, since the string given is completely
arbitrary. But someone who knows more about the internal structure of code
objects may give you better tips.
[color=blue]
> I also tried pydb.py using IPython 0.7.0 (using an automatically
> updated Fedora Core 5 RPM) and although I was expecting pydb.py breakage
> from what you report, I didn't find any difference.
>
>
> % ipython
> Python 2.4.2 (#1, Dec 17 2005, 13:02:22)
> Type "copyright" , "credits" or "license" for more information.
>
> IPython 0.7.0 -- An enhanced Interactive Python.
> ? -> Introduction to IPython's features.
> %magic -> Information about IPython's 'magic' % functions.
> help -> Python's own help system.
> object? -> Details about 'object'. ?object also works, ?? prints more.
>
> In [1]: %run /usr/lib/python2.4/site-packages/pydb.py test1.py[color=green]
>> /home/rocky/python/test1.py(2)?()[/color]
> -> import sys
> (Pydb) where
> -> 0 in file '/home/rocky/python/test1.py' at line 2
> ## 1 in exec cmd in globals, locals at line 1
> ## 2 run() called from file '/usr/lib/python2.4/bdb.py' at line 366
> (Pydb)
>
> Note entry ##1 is *not*
> File "<string>", line 1, in ?
>
> So somehow I guess ipython is not intercepting or changing how compile
> was run here.[/color]

Oh, that's because you're using %run, so your code is in complete control.
What I meant about a restriction was thiking about using your pydb as the
debugger called by ipython when exceptions occur.

Try

%pdb

and then %run any script with an error in it, to be dropped in post-mortem
mode inside the stack. That uses an ipython-enhanced pdb, but not your
pydb. As your code matures and improves, it would be nice to, for example,
make it possible to plug it into ipython as a 'better debugger'. But
there, you'll see '<ipython console>' as the filename markers.

Anyway, good luck with this. If you are interested in ipython integration,
I suggest the ipython-dev list as a better place for discussion: I only
monitor c.l.py on occasion, so I may well miss things here.

Regards,

f

[Ziga Seilnacht]

Re: Getting better traceback info on exec and execfile - introspection?

R. Bernstein wrote:
..
..
..[color=blue]
> which is perhaps is a little more honest since one is not really in a
> file called <string>. However the way the debugger gets this *is*
> still a little hoaky in that it looks for something in the frame's
> f_code.co_filen ame *called* <string>. And from that it *assumes* this
> is an exec, so it can't really tell the difference between an exec
> command an execfile command, or a file called <string>. But I suppose
> a little more hoakiness *could* be added to look at the outer frame
> and parse the source line for "exec" or "execfile".[/color]

You should check the getFrameInfo function in zope.interface package:

http://svn.zope.org/Zope3/trunk/src/zope/interface/advice.py?rev=25177&view=markup

[color=blue]
> And suppose instead of '<string>' I'd like to give the value or the
> leading prefix of the value instead of the unhelpful word '<string>'?
> How would one do that? Again, one way is to go into the outer frame
> get the source line (if that exists), parse that and interpolate
> argument to exec(file). Is there are better way?[/color]

Py library (http://codespeak.net/py/current/doc/index.html) has some
similar functionality in the code subpackage.

[R. Bernstein]

Re: Getting better traceback info on exec and execfile - introspection?

"Ziga Seilnacht" <ziga.seilnacht @gmail.com> writes:[color=blue]
> You should check the getFrameInfo function in zope.interface package:
> http://svn.zope.org/Zope3/trunk/src/...77&view=markup[/color]

Thanks! Just looked at that. The logic in the relevant part (if I've extracted
this correctly):

f_globals = frame.f_globals
hasName = '__name__' in f_globals
module = hasName and sys.modules.get (f_globals['__name__']) or None
namespaceIsModu le = module and module.__dict__ is f_globals
if not namespaceIsModu le:
# some kind of funky exec
kind = "exec"

is definitely not obvious to me. exec's don't have module namespace
(or something or other)? Okay. And that the way to determine the
module-namespace thingy is whatever that logic is? Are the assumptions
here be likely to be valid in the future?

Another problem I have with that code is that it uses the Zope Public
License. But the code is adapted from the Python Enterprise
Application Kit (PEAK) which doesn't seem to use the Zope Public
License. I'm not sure it's right to adopt a Zope Public License just
for this.

So instead, I followed the other avenue I suggested which is
dissembling the statement around the frame. Talk about the kettle
calling the pot black! Yes, I don't know if the assumptions in this
method are likely to be valid in the future either.

But I can use op-code examination to also help me determine if we are
stopped at a def statement which I want to skip over. So here's the
code I've currently settled on:

from opcode import opname
def op_at_frame(fra me):
code = frame.f_code.co _code
pos = frame.f_lasti
op = ord(code[pos])
return opname[op]

def is_exec_stmt(fr ame):
"""Return True if we are looking at an exec statement"""
return frame.f_back is not None and op_at_frame(fra me.f_back)=='EX EC_STMT'

re_def = re.compile(r'^\ s*def\s+')
def is_def_stmt(lin e, frame):
"""Return True if we are looking at a def statement"""
# Should really also check that the operand is a code object
return re_def.match(li ne) and op_at_frame(fra me)=='LOAD_CONS T'

But it may just be because I wrote it that I find it easier to
understand and more straightforward to fathom.
[color=blue][color=green]
> > And suppose instead of '<string>' I'd like to give the value or the
> > leading prefix of the value instead of the unhelpful word '<string>'?
> > How would one do that? Again, one way is to go into the outer frame
> > get the source line (if that exists), parse that and interpolate
> > argument to exec(file). Is there are better way?[/color]
>
> Py library (http://codespeak.net/py/current/doc/index.html) has some
> similar functionality in the code subpackage.[/color]

Again, many thanks. I'll have to study this further. It may be that
exec and so on are wrapped so that it's possible to squirrel away the
string before calling exec. Again, dunno. But thanks for the pointer.

[R. Bernstein]

Re: Getting better traceback info on exec and execfile - introspection?

Fernando Perez <fperez.net@gma il.com> writes:[color=blue]
> I thought a little about this. One possibility ...[/color]

Thanks. A sibling thread has the code I'm currently using.
[color=blue]
> Oh, that's because you're using %run, so your code is in complete control.
> What I meant about a restriction ...[/color]
Okay.
[color=blue]
> If you are interested in ipython integration,[/color]

Yes I am.
[color=blue]
> I suggest the ipython-dev list as a better place for discussion: I only
> monitor c.l.py on occasion, so I may well miss things here.[/color]

Okay now subscribed. But interestingly I looked in the IPython PDF and
didn't see mention of it when looking for contact info. I do however
see it (now) listed on http://ipython.scipy.org/. Thanks again.

[R. Bernstein]

Re: Getting better traceback info on exec and execfile - introspection?

Fernando Perez <fperez.net@gma il.com> writes:[color=blue]
> So any regexp-matching based approach here is likely to be fairly brittle,
> unless you restrict your tool to the standard python interpreter, and you
> get some guarantee that it will always tag interactive code with
> '<string>'.[/color]

Meant to mention for what it's worth. Looks like I'm not the first to use the
filename == '<string>' test. I note this in the stock pdb.py:

# To be overridden in derived debuggers
def defaultFile(sel f):
"""Produce a reasonable default."""
filename = self.curframe.f _code.co_filena me
if filename == '<string>' and self.mainpyfile :
filename = self.mainpyfile
return filename

I'm not sure under what conditions this equality test normally occurs
though.