Using python for _large_ projects like IDE

[Sridhar R]

Hi,

I am a little experienced python programmer (2 months). I am somewhat
experienced in C/C++. I am planning (now in design stage) to write an
IDE in python. The IDE will not be a simple one. I had an idea of
writing the IDE in C/C++, as it is a big project, bcoz of the
following

1. if python is used, then the memory required for running the IDE
will be high.
2. if pure python is used, the IDE will be slower.

I'm not sure whether they are true.

Then I thought of using python for 90% of the code and using C for
other performance critical part. But the problem is in identifying
the performance critical code. People suggest to start with pure
python and then _recode_ the performance critical code after profiling
the original code. But I fear whether there will be a conflit in say
data structures. Not yet expert in extending/embedding python.
Are there any ways to forsee the performance critical parts?

Also, any suggestions on reducing memory usage of big applications
written in python?

[Rene Pijlman]

Re: Using python for _large_ projects like IDE

Sridhar R:[color=blue]
>I am a little experienced python programmer (2 months). [...]
>I am planning (now in design stage) to write an IDE[/color]

I've never developed an IDE, but it wouldn't be my first attempt to write
an app. It's huge and complex.

I started with prime numbers, tic-tac-toe, towers of hanoi, traveling
sales man, merge sort, and little things like that :-)
[color=blue]
>Are there any ways to forsee the performance critical parts?[/color]

Sure, no problem. Post the design please.
[color=blue]
>Also, any suggestions on reducing memory usage of big applications
>written in python?[/color]

Only the obvious: don't keep more data alive than strictly necessary.

--
René Pijlman

[Samuel Walters]

Re: Using python for _large_ projects like IDE

| Sridhar R said |[color=blue]
> 1. if python is used, then the memory required for running the IDE
> will be high.
> 2. if pure python is used, the IDE will be slower.
>
> I'm not sure whether they are true.[/color]

Well, it depends on your definition of "high memory usage." I had no
problems running GUI python apps on my old pentium 100mhz with 74 megs of
ram. So, unless your program should be able to run on smaller systems, I
wouldn't worry much.

[color=blue]
> Then I thought of using python for 90% of the code and using C for
> other performance critical part. But the problem is in identifying
> the performance critical code. People suggest to start with pure
> python and then _recode_ the performance critical code after profiling
> the original code. But I fear whether there will be a conflit in say
> data structures. Not yet expert in extending/embedding python.
> Are there any ways to forsee the performance critical parts?[/color]

Data structures are easier to handle than you'd think when extending
python. I recommend either boost or pyrex. I've used pyrex, but most
people seem to use boost.

There are great python profiling tools. It's not at all difficult to find
the performance critical portions.

As far as advice on forseeing performance critical parts, keep in mind
that there are different standards for different portions of a program.
An IDE would have many parts where interactivity is a concern. Noone
likes to see their keystrokes appear two seconds after they have been
typed. Things like syntax highlighting can form a bottleneck, but I've
seen enough apps that manipulate data like this that I know it can be
done.

There are other parts, such as search-and-replace that people are a bit
more patient about. Try to separate the two types of functionality so
that they can be recoded independently.

HTH

Sam Walters.

--
Never forget the halloween documents.

http://www.opensource.org/halloween/

""" Where will Microsoft try to drag you today?
Do you really want to go there?"""

[Izzie]

Re: Using python for _large_ projects like IDE

Sridhar R wrote:
[color=blue]
> I am a little experienced python programmer (2 months). I am somewhat
> experienced in C/C++. I am planning (now in design stage) to write an
> IDE in python. The IDE will not be a simple one. I had an idea of
> writing the IDE in C/C++, as it is a big project, bcoz of the
> following[/color]

Don't bother, there are many good IDEs that can be used with python,
there are even more excellent text editors that (IMO) are better
than all IDEs that I tried. Why waste you time?

i.

[DilbertFan]

Re: Using python for _large_ projects like IDE

If you do write this IDE, please include a compiler warning that catches
the missing () at the end of a function that doesn't have arguments. I've
sunk into a dark world of a diabolically mind-destroying hysteria while
trying to troubleshoot some scripts, and then realizing that a python
function was not executing because I was calling it without '()'. But
Python doesn't complain, or give you a message or anything,... it simply
doesn't execute that function!

Big reason I forget the empty parenthesis: I do Delphi also, and it doesn't
require it.

(Meanwhile, for Python I'm using Komodo, which is really good. Funny how I
keep upgrading UltraEdit when I go ahead and register more language-specific
tools anyway )

"Sridhar R" <sridharinfinit y@yahoo.com> wrote in message
news:930ba99a.0 401180625.5863a cd4@posting.goo gle.com...[color=blue]
> Hi,
>
> I am a little experienced python programmer (2 months). I am somewhat
> experienced in C/C++. I am planning (now in design stage) to write an
> IDE in python. The IDE will not be a simple one. I had an idea of
> writing the IDE in C/C++, as it is a big project, bcoz of the
> following
>
> 1. if python is used, then the memory required for running the IDE
> will be high.
> 2. if pure python is used, the IDE will be slower.
>
> I'm not sure whether they are true.
>
> Then I thought of using python for 90% of the code and using C for
> other performance critical part. But the problem is in identifying
> the performance critical code. People suggest to start with pure
> python and then _recode_ the performance critical code after profiling
> the original code. But I fear whether there will be a conflit in say
> data structures. Not yet expert in extending/embedding python.
> Are there any ways to forsee the performance critical parts?
>
> Also, any suggestions on reducing memory usage of big applications
> written in python?[/color]

[djw]

Re: Using python for _large_ projects like IDE

I would take a look at Eric. Its written in Python, is fully featured
and is plenty fast. In fact, there are plenty of IDEs written in Python
or fo Python, why would you want to create yet another one? Also, as a
beginning Python programmer, I fear you may be getting in over your head
(I'm a 2+ year Python and 10+ year C/C++ developer and would be hesitant
to tackle an IDE).

Anyway, Eric is here:

http://www.die-offenbachs.de/detlev/eric3.html

-Don



Sridhar R wrote:
[color=blue]
> Hi,
>
> I am a little experienced python programmer (2 months). I am somewhat
> experienced in C/C++. I am planning (now in design stage) to write an
> IDE in python. The IDE will not be a simple one. I had an idea of
> writing the IDE in C/C++, as it is a big project, bcoz of the
> following
>
> 1. if python is used, then the memory required for running the IDE
> will be high.
> 2. if pure python is used, the IDE will be slower.
>
> I'm not sure whether they are true.
>
> Then I thought of using python for 90% of the code and using C for
> other performance critical part. But the problem is in identifying
> the performance critical code. People suggest to start with pure
> python and then _recode_ the performance critical code after profiling
> the original code. But I fear whether there will be a conflit in say
> data structures. Not yet expert in extending/embedding python.
> Are there any ways to forsee the performance critical parts?
>
> Also, any suggestions on reducing memory usage of big applications
> written in python?[/color]

[Samuel Walters]

Re: Using python for _large_ projects like IDE

| DilbertFan said |
[color=blue]
> If you do write this IDE, please include a compiler warning that catches
> the missing () at the end of a function that doesn't have arguments. I've
> sunk into a dark world of a diabolically mind-destroying hysteria while
> trying to troubleshoot some scripts, and then realizing that a python
> function was not executing because I was calling it without '()'. But
> Python doesn't complain, or give you a message or anything,... it simply
> doesn't execute that function![/color]

Try PyChecker. It should warn you about such things.

http://pychecker.sourceforge.net/

HTH

Sam Walters.

--
Never forget the halloween documents.

http://www.opensource.org/halloween/

""" Where will Microsoft try to drag you today?
Do you really want to go there?"""

[DilbertFan]

Re: Using python for _large_ projects like IDE

Thanks Sam
"Samuel Walters" <swalters_usene t@yahoo.com> wrote in message
news:pan.2004.0 1.18.22.01.46.6 70164@yahoo.com ...[color=blue]
> | DilbertFan said |
>[color=green]
> > If you do write this IDE, please include a compiler warning that[/color][/color]
catches[color=blue][color=green]
> > the missing () at the end of a function that doesn't have arguments.[/color][/color]
I've[color=blue][color=green]
> > sunk into a dark world of a diabolically mind-destroying hysteria while
> > trying to troubleshoot some scripts, and then realizing that a python
> > function was not executing because I was calling it without '()'. But
> > Python doesn't complain, or give you a message or anything,... it simply
> > doesn't execute that function![/color]
>
> Try PyChecker. It should warn you about such things.
> http://pychecker.sourceforge.net/
>
> HTH
>
> Sam Walters.
>
> --
> Never forget the halloween documents.
> http://www.opensource.org/halloween/
> """ Where will Microsoft try to drag you today?
> Do you really want to go there?"""
>[/color]

[Eric Brunel]

Re: Using python for _large_ projects like IDE

Sridhar R wrote:[color=blue]
> Hi,
>
> I am a little experienced python programmer (2 months). I am somewhat
> experienced in C/C++. I am planning (now in design stage) to write an
> IDE in python. The IDE will not be a simple one. I had an idea of
> writing the IDE in C/C++, as it is a big project, bcoz of the
> following
>
> 1. if python is used, then the memory required for running the IDE
> will be high.[/color]

It's quite hard to tell, since I've never seen anybody writing a significant
application *both* in C/C++ and Python to compare their memory usage... I'd say
memory usage depends a lot more on the design than on the language used. Python
does introduce an extra cost in memory consumption, but the larger the
application, the less you'll notice it.
[color=blue]
> 2. if pure python is used, the IDE will be slower.[/color]

An application coded in Python will actually be slower than the same application
coded in C/C++. I won't repeat what others have already said about what *looks*
slow to a user.
[color=blue]
> I'm not sure whether they are true.
>
> Then I thought of using python for 90% of the code and using C for
> other performance critical part. But the problem is in identifying
> the performance critical code. People suggest to start with pure
> python and then _recode_ the performance critical code after profiling
> the original code. But I fear whether there will be a conflit in say
> data structures. Not yet expert in extending/embedding python.[/color]

We did just that for quite a big application and it works like a charm: the
critical code is usually one that does many calculations and it's usually quite
easy to isolate this part in a module handling only base types. And even if it's
not true, as somebody already said, it's easier to handle Python objects at C
level that it may seem at first sight.
[color=blue]
> Are there any ways to forsee the performance critical parts?[/color]

I really think you shouldn't care; remember, "premature optimization is the root
of all evil" ;-)
[color=blue]
> Also, any suggestions on reducing memory usage of big applications
> written in python?[/color]

Design them keeping this problem in mind if you really have to. But...
"premature optimization ...". You'll always be able to make compromises on a
well designed application afterwards to reduce memory usage; it's indeed a lot
easier than to maintain an application originally badly designed for bad reasons
(e.g. reduce memory consumption).

HTH
--
- Eric Brunel <eric dot brunel at pragmadev dot com> -
PragmaDev : Real Time Software Development Tools - http://www.pragmadev.com