efficient way for building class browsers.

[Sridhar R]

I am looking for a class browser that has these features.

1. Given a symbol (class, method or function) it should giveback
the lineno n source code
2. It should be efficient and quick.

I looked at the `pyclbr` module, but it's slower. I will be _often_
regenerating the symbols from exactly one python source file. But for
an average python script in my machine it takes 2 secs for pyclbr to
build the dictionary of symbols.

When I profiled pyclbr, I found that most of the time was spent on the
tokenize.get_to ken** functions, and the remaining time was spent on
pyclbr._readmod ule function.

Remember I need a class browser, that also tells me the line no in
source code of that symbol. (So compiler module (ast) can't be used -
Is it so? )

Are there any better ways for achieving this?

[Josiah Carlson]

Re: efficient way for building class browsers.

> I am looking for a class browser that has these features.

Only a browser? Text or GUI? What OS? Any particular GUI toolkit?
[color=blue]
> Remember I need a class browser, that also tells me the line no in
> source code of that symbol. (So compiler module (ast) can't be used -
> Is it so? )[/color]

compiler.ast can be used for this, but only produces proper results when
the file has no syntax errors. Don't get me wrong, using the output of
compiler.ast can be a pain in the ass, and slow as hell (5 seconds for
100k of source on a celeron 400), but it can be used.
[color=blue]
> Are there any better ways for achieving this?[/color]

You can write a limited parser (subject to certain flaws), one that
doesn't use tokenize or compiler.ast, and get pretty fast results. I
wrote one to go into my own editor PyPE (pype.sourcefor ge.net), and it
has no problems parsing a 100k Python source file and displaying the
source tree in a GUI in less than 1/2 second on a celeron 400.

It wouldn't be too bad to pull the tree portion out, what kind of
browser do you need?

- Josiah

[Lothar Scholz]

Re: efficient way for building class browsers.

Josiah Carlson <jcarlson@nospa m.uci.edu> wrote in message news:<c04nvs$h6 j$1@news.servic e.uci.edu>...[color=blue][color=green]
> > I am looking for a class browser that has these features.[/color]
>
> Only a browser? Text or GUI? What OS? Any particular GUI toolkit?
>[color=green]
> > Remember I need a class browser, that also tells me the line no in
> > source code of that symbol. (So compiler module (ast) can't be used -
> > Is it so? )[/color]
>
> compiler.ast can be used for this, but only produces proper results when
> the file has no syntax errors. Don't get me wrong, using the output of
> compiler.ast can be a pain in the ass, and slow as hell (5 seconds for
> 100k of source on a celeron 400), but it can be used.
>[color=green]
> > Are there any better ways for achieving this?[/color]
>
> You can write a limited parser (subject to certain flaws), one that
> doesn't use tokenize or compiler.ast, and get pretty fast results. I
> wrote one to go into my own editor PyPE (pype.sourcefor ge.net), and it
> has no problems parsing a 100k Python source file and displaying the
> source tree in a GUI in less than 1/2 second on a celeron 400.[/color]

Thats slow i have a parser that parses 100k in less then 15 ms on a
1,2 GHz system. It will be part of "http://www.python-ide.com"
It is a 100% GNU Eiffel solution which generates plain C code.

I think the 1/2 second could also be done with some intelligent
regexpr's. There is something like this in the IDLE code.

[Josiah Carlson]

Re: efficient way for building class browsers.

>>You can write a limited parser (subject to certain flaws), one that[color=blue][color=green]
>>doesn't use tokenize or compiler.ast, and get pretty fast results. I
>>wrote one to go into my own editor PyPE (pype.sourcefor ge.net), and it
>>has no problems parsing a 100k Python source file and displaying the
>>source tree in a GUI in less than 1/2 second on a celeron 400.[/color]
>
> Thats slow i have a parser that parses 100k in less then 15 ms on a
> 1,2 GHz system. It will be part of "http://www.python-ide.com"
> It is a 100% GNU Eiffel solution which generates plain C code.[/color]

I never said it was the fastest, I said it was "pretty fast", and
provided results that it was faster than the pyclbr and compiler.ast
modules.

It is also written in Python, which seems to be more of what Sridhar was
looking for. Furthermore, the source for the fast parser is freely
available. Considering that Arachno Python IDE is a commercial effort,
the liklihood of the parser being available in source or loadable module
format seems fairly unlikely.

[color=blue]
> I think the 1/2 second could also be done with some intelligent
> regexpr's. There is something like this in the IDLE code.[/color]

Checking the IDLE source for version 1.0 (I would check CVS, but CVS
seems to be having issues at sourceforge), it uses pyclbr.

- Josiah

[Michele Simionato]

Re: efficient way for building class browsers.

Josiah Carlson <jcarlson@nospa m.uci.edu> wrote in message news:<c062lt$mb 3$1@news.servic e.uci.edu>...[color=blue][color=green]
> > I think the 1/2 second could also be done with some intelligent
> > regexpr's. There is something like this in the IDLE code.[/color]
>
> Checking the IDLE source for version 1.0 (I would check CVS, but CVS
> seems to be having issues at sourceforge), it uses pyclbr.
>[/color]

The inspect module uses regexps to extract the source code from classes
(and it is always a very good reading).

Michele

[Sridhar R]

Re: efficient way for building class browsers.

Josiah Carlson <jcarlson@nospa m.uci.edu> wrote in message news:<c04nvs$h6 j$1@news.servic e.uci.edu>...[color=blue][color=green]
> > I am looking for a class browser that has these features.[/color]
>
> Only a browser? Text or GUI? What OS? Any particular GUI toolkit?[/color]

IDE. Linux,Windows,M ac. PyGTK
[color=blue][color=green]
> > Remember I need a class browser, that also tells me the line no in
> > source code of that symbol. (So compiler module (ast) can't be used -
> > Is it so? )[/color]
>
> compiler.ast can be used for this, but only produces proper results when
> the file has no syntax errors. Don't get me wrong, using the output of
> compiler.ast can be a pain in the ass, and slow as hell (5 seconds for
> 100k of source on a celeron 400), but it can be used.
>[color=green]
> > Are there any better ways for achieving this?[/color]
>
> You can write a limited parser (subject to certain flaws), one that
> doesn't use tokenize or compiler.ast, and get pretty fast results. I
> wrote one to go into my own editor PyPE (pype.sourcefor ge.net), and it
> has no problems parsing a 100k Python source file and displaying the
> source tree in a GUI in less than 1/2 second on a celeron 400.
>[/color]

I'll try that. I have come across tagmanager (C) code written for
the Anjuta (http://anjuta.org) project. I should give it a try too.
[color=blue]
> It wouldn't be too bad to pull the tree portion out, what kind of
> browser do you need?[/color]

I will put it in brief. The class browser can't support all
standard and third party python packages, since most of them is
available as shared objects. So mine is focused only on the current
project, i.e. user created (or application) python files. So the
browser should be able to fetch the tags (class, functions, methods)
from source code along with the line number information. That's all
about. Any further possiblities in this regard is also expected.
[color=blue]
> - Josiah[/color]

[Josiah Carlson]

Re: efficient way for building class browsers.

>>>I am looking for a class browser that has these features.[color=blue][color=green]
>>
>>Only a browser? Text or GUI? What OS? Any particular GUI toolkit?[/color]
>
> IDE. Linux,Windows,M ac. PyGTK[/color]

Seemingly this suggests that you want an IDE that runs on linux,
windows, mac, and that uses the PyGTK toolkit.

I personally don't spend much time looking through PyGTK apps, so I
can't be of much help. On the other hand, if you have a parser,
constructing the tree control/widget from the parsing is very easy.

[color=blue][color=green]
>>You can write a limited parser (subject to certain flaws), one that
>>doesn't use tokenize or compiler.ast, and get pretty fast results. I
>>wrote one to go into my own editor PyPE (pype.sourcefor ge.net), and it
>>has no problems parsing a 100k Python source file and displaying the
>>source tree in a GUI in less than 1/2 second on a celeron 400.[/color]
>
> I'll try that. I have come across tagmanager (C) code written for
> the Anjuta (http://anjuta.org) project. I should give it a try too.[/color]

The control in PyPE is written for wxPython, but the parser is platform
independant.

[color=blue]
> I will put it in brief. The class browser can't support all
> standard and third party python packages, since most of them is
> available as shared objects. So mine is focused only on the current
> project, i.e. user created (or application) python files. So the
> browser should be able to fetch the tags (class, functions, methods)
> from source code along with the line number information. That's all
> about. Any further possiblities in this regard is also expected.[/color]

Fetching the various function/class/method names is easy, look at the
parser included with PyPE. It isn't perfect, but works pretty well
considering that it is neither a regular expression, nor does it have a
tokenizer.

- Josiah