Python for large projects

[assaf__@walla.com]

Hello,

I am beginning to work on a fairly large project and I'm considering to use python for most of the coding, but I need to make sure first that it is reliable enough.

I need to make sure that I won't have surprises when my program runs on different real-world systems. So far I wrote a little script with python using urllib, and on one computer it failed completely because of a problem in getting the proxies (in my opinion this is a bug). How likely are such things to happen and how often, and to what extent are they more prevalent in python in comparison to C++?

If python is indeed suitable for large projects, how common is it to actually use if for such purposes? Is there perhaps a list of examples of real projects using python?

Thanks,

Bob

-----------------------------------------------------------------------
Walla! Mail, Get Your Private, Free E-mail from Walla! at:

http://mail.walla.co.il

[Andrew Wilkinson]

Re: Python for large projects

assaf__@walla.c om wrote:[color=blue]
> If python is indeed suitable for large projects, how common is it to
> actually use if for such purposes? Is there perhaps a list of examples of
> real projects using python?[/color]

I would say that Python is very suitable for use in large projects, just the
benefit of automatic memory management is enough to convince me that it
beats C++. The powerful builtin types are another big win - what you would
spend ages coding in C++ is often a no brainer in Python.

Obviously there are downsides, I find the lack of static type checking to
cause some bugs to hide in my code that in other languages would be found
at compile time. In general (and IMHO) though, the pros far outway cons.

There is a decent sized list of companies that use Python at
http://pythonology.org/success.

HTH,
Andrew

[Paul Rubin]

Re: Python for large projects

assaf__@walla.c om writes:[color=blue]
> I am beginning to work on a fairly large project and I'm considering
> to use python for most of the coding, but I need to make sure first
> that it is reliable enough.
>
> I need to make sure that I won't have surprises when my program runs
> on different real-world systems. So far I wrote a little script with
> python using urllib, and on one computer it failed completely
> because of a problem in getting the proxies (in my opinion this is a
> bug). How likely are such things to happen and how often, and to
> what extent are they more prevalent in python in comparison to C++?[/color]

The Python language is in general well-designed and much more concise
than C++. A big program in C++ may map to a much smaller Python
program, turning a large project into a small project, and making it
less relevant whether Python works for large projects.

Python's library does have a lot of small gaps like the one you found
in urllib. As the Twisted Matrix documentation puts it, you find
yourself re-inventing the wheel a lot, because you discover that the
existing wheels are often square and made of glue.
[color=blue]
> If python is indeed suitable for large projects, how common is it to
> actually use if for such purposes? Is there perhaps a list of
> examples of real projects using python?[/color]

The canonical example of a complex Python project is Zope
(www.zope.com). It's medium sized by the standards of big C++
projects, but as mentioned, a medium amount of Python code can
implement functionality that would take a much larger amount of C++.

Because Python is interpreted and highly dynamic, Python programs tend
to run slower than comparable C++ programs. Whether that's a problem
for you depends on your application. If you have specific Python
functions that are bottlenecks, you can re-implement them in C and
call them through Python's C API. There's also a semi-experimental
native-code Python compiler called Psyco that produces a considerable
speedup at the cost of increased memory consumption. The
next-generation Python implementation (PyPy or Python in Python) will
reportedly use Psyco or something simliar, in a more fundamental way.

In short, there's not a quick and simple answer to your question of
whether Python is right for what you're doing. It's great for lots of
things, not so hot for some others, and is still evolving rather
quickly, so an unsuitable application today may become suitable in a
forthcoming release.

[Cameron Laird]

Re: Python for large projects

In article <7xisgxt51r.fsf @ruckus.brouhah a.com>,
Paul Rubin <http://phr.cx@NOSPAM.i nvalid> wrote:[color=blue]
>assaf__@walla. com writes:[color=green]
>> I am beginning to work on a fairly large project and I'm considering
>> to use python for most of the coding, but I need to make sure first
>> that it is reliable enough.
>>
>> I need to make sure that I won't have surprises when my program runs
>> on different real-world systems. So far I wrote a little script with
>> python using urllib, and on one computer it failed completely
>> because of a problem in getting the proxies (in my opinion this is a
>> bug). How likely are such things to happen and how often, and to
>> what extent are they more prevalent in python in comparison to C++?[/color]
>
>The Python language is in general well-designed and much more concise
>than C++. A big program in C++ may map to a much smaller Python
>program, turning a large project into a small project, and making it
>less relevant whether Python works for large projects.
>
>Python's library does have a lot of small gaps like the one you found
>in urllib. As the Twisted Matrix documentation puts it, you find
>yourself re-inventing the wheel a lot, because you discover that the
>existing wheels are often square and made of glue.
>[color=green]
>> If python is indeed suitable for large projects, how common is it to
>> actually use if for such purposes? Is there perhaps a list of
>> examples of real projects using python?[/color]
>
>The canonical example of a complex Python project is Zope
>(www.zope.com). It's medium sized by the standards of big C++
>projects, but as mentioned, a medium amount of Python code can
>implement functionality that would take a much larger amount of C++.
>
>Because Python is interpreted and highly dynamic, Python programs tend
>to run slower than comparable C++ programs. Whether that's a problem
>for you depends on your application. If you have specific Python
>functions that are bottlenecks, you can re-implement them in C and
>call them through Python's C API. There's also a semi-experimental
>native-code Python compiler called Psyco that produces a considerable
>speedup at the cost of increased memory consumption. The
>next-generation Python implementation (PyPy or Python in Python) will
>reportedly use Psyco or something simliar, in a more fundamental way.
>
>In short, there's not a quick and simple answer to your question of
>whether Python is right for what you're doing. It's great for lots of
>things, not so hot for some others, and is still evolving rather
>quickly, so an unsuitable application today may become suitable in a
>forthcoming release.[/color]

You might also have occasion to learn about Pyrex and
.... well, I'm in the Trotskyite wing on this question.
It's not just that Python can work for large projects.
I sincerely regard it as an even *better* comparative
choice on large projects; C++ and Java, the usual com-
petition, show all sorts of blemishes when one scales
the size of the project. Python remains usable, even
at the high end.
--

Cameron Laird <claird@phaseit .net>
Business: http://www.Phaseit.net

[Paul Rubin]

Re: Python for large projects

claird@lairds.c om (Cameron Laird) writes:[color=blue]
> You might also have occasion to learn about Pyrex and ... well, I'm
> in the Trotskyite wing on this question. It's not just that Python
> can work for large projects. I sincerely regard it as an even
> *better* comparative choice on large projects; C++ and Java, the
> usual com- petition, show all sorts of blemishes when one scales the
> size of the project. Python remains usable, even at the high end.[/color]

I keep hearing this, but I don't see any large (much less very large)
applications that have been done in Python; Zope is medium sized.

Suppose you wanted to write any of the following:

1) Optimizing C/C++ compiler, like GCC
2) Full featured web browser, like MSIE or Mozilla
3) Full featured office suite, like MS Office or Open Office or KDE
4) Avionics for the space shuttle
5) Internals of a large telephone/data switch
6) Tax processing software for the IRS
7) Operating system kernel (Linux: the next generation)
8) Accounting software for a big bank
9) Full featured database like Oracle or Postgres
10) ... well you get the idea.

Which, if any, would you write in Python? By "write in Python" I mean
the central framework and most of the code is written in Python,
though you're allowed to use the C API as needed. Using Python to
provide some functions around the edges (e.g. the operator UI for the
phone switch) doesn't count. Note that these are all supposed to be
used in production and not just as technology demos, so speed matters
(nobody wants a compiler that's 10x slower than GCC. GCC is slow
enough already).

I think I'd spend some time at least considering Python in each of the
above cases, but I'm not sure I could convincingly make it fly for any
of them.

[Diez B. Roggisch]

Re: Python for large projects

Paul Rubin wrote:
[color=blue]
> 1) Optimizing C/C++ compiler, like GCC
> 2) Full featured web browser, like MSIE or Mozilla
> 3) Full featured office suite, like MS Office or Open Office or KDE
> 4) Avionics for the space shuttle
> 5) Internals of a large telephone/data switch
> 6) Tax processing software for the IRS
> 7) Operating system kernel (Linux: the next generation)
> 8) Accounting software for a big bank
> 9) Full featured database like Oracle or Postgres
> 10) ... well you get the idea.
>
> Which, if any, would you write in Python? By "write in Python" I mean[/color]

Which, if any, would you write in JAVA or C#? I think for a project in
"userland" thats not on the number crunching or machine architecture side
like a kernel2, python often would be a viable alternative.

And the spaceshuttle runs on a 68K processor - I bet a modern 3GHz processor
running python can beat that :)


--
Regards,

Diez B. Roggisch

[Heather Coppersmith]

Re: Python for large projects

On 22 Mar 2004 15:37:27 -0800,
Paul Rubin <http://phr.cx@NOSPAM.i nvalid> wrote:
[color=blue]
> 1) Optimizing C/C++ compiler, like GCC
> 2) Full featured web browser, like MSIE or Mozilla
> 3) Full featured office suite, like MS Office or Open Office or KDE
> 4) Avionics for the space shuttle
> 5) Internals of a large telephone/data switch
> 6) Tax processing software for the IRS
> 7) Operating system kernel (Linux: the next generation)
> 8) Accounting software for a big bank
> 9) Full featured database like Oracle or Postgres
> 10) ... well you get the idea.[/color]
[color=blue]
> Which, if any, would you write in Python? ... Note that these
> are all supposed to be used in production and not just as
> technology demos, so speed matters ...[/color]

With the "speed counts" caveat, that's a bit of a trick question,
trivially answered with None Of The Above (okay, so maybe most
bankers (application 8) are in less of a hurry than most user
processes (application 7)).

With sufficiently powerful hardware and sufficiently patient
clients, I'd take Python over C for any of those projects except:

Life/death applications, like 4, and maybe 5, and maybe 7
(depends on who's using the OS and for what). Python is not
well-specified enough (too many things depend on the
underlying C library) for this type of application.

Hardware-oriented applications, like maybe parts of 4 and 5
(depending on what the interfaces to the hardware look like),
and the low-level pieces of 7.

Specifically, I can't think of a single advantage C has over
Python (except speed and footprint) for applications 1, 2, 3, 5,
6, 8, and 9.

What advantages *do* C and Java have for such large projects? How
many bug-free, meets-the-full-spec, on-time, under-budget examples
of applications written in those languages do we have?

Regards,
Heather

--
Heather Coppersmith
That's not right; that's not even wrong. -- Wolfgang Pauli

[Jay O'Connor]

Re: Python for large projects

Paul Rubin wrote:[color=blue]
> claird@lairds.c om (Cameron Laird) writes:
>[color=green]
>>You might also have occasion to learn about Pyrex and ... well, I'm
>>in the Trotskyite wing on this question. It's not just that Python
>>can work for large projects. I sincerely regard it as an even
>>*better* comparative choice on large projects; C++ and Java, the
>>usual com- petition, show all sorts of blemishes when one scales the
>>size of the project. Python remains usable, even at the high end.[/color]
>
>
> I keep hearing this, but I don't see any large (much less very large)
> applications that have been done in Python; Zope is medium sized.
>
> Suppose you wanted to write any of the following:
>
> 1) Optimizing C/C++ compiler, like GCC
> 2) Full featured web browser, like MSIE or Mozilla
> 3) Full featured office suite, like MS Office or Open Office or KDE
> 4) Avionics for the space shuttle
> 5) Internals of a large telephone/data switch
> 6) Tax processing software for the IRS
> 7) Operating system kernel (Linux: the next generation)
> 8) Accounting software for a big bank
> 9) Full featured database like Oracle or Postgres
> 10) ... well you get the idea.
>
> Which, if any, would you write in Python?[/color]

6 and 8, maybe 3...and 10 of course

Take care,
Jay

[Paul Rubin]

Re: Python for large projects

Heather Coppersmith <me@privacy.net > writes:[color=blue]
> What advantages *do* C and Java have for such large projects? How
> many bug-free, meets-the-full-spec, on-time, under-budget examples
> of applications written in those languages do we have?[/color]

It was a real question, not a rhetorical one, and your answers are
reasonable.

We may not have C or Java examples of those apps that are on-time and
under budget, but at least we have examples now that are deployed and
that work, even if they were delivered late and over budget. We don't
have ANY examples for programs like that in Python, whether delivered
on time or not. On the other hand, we haven't been writing stuff like
that in Python for as long.

I'd like to think that a suitable implementation of Python, or
something pretty closely resembling Python as we know it, could do
pretty much anything Common Lisp can do, and CL has been used for some
very big apps. However, the current Python implementations seem aimed
at smaller projects.

[Paul Rubin]

Re: Python for large projects

Jay O'Connor <jayoconnor@ear thlink.net> writes:[color=blue][color=green]
> > 1) Optimizing C/C++ compiler, like GCC
> > 2) Full featured web browser, like MSIE or Mozilla
> > 3) Full featured office suite, like MS Office or Open Office or KDE
> > 4) Avionics for the space shuttle
> > 5) Internals of a large telephone/data switch
> > 6) Tax processing software for the IRS
> > 7) Operating system kernel (Linux: the next generation)
> > 8) Accounting software for a big bank
> > 9) Full featured database like Oracle or Postgres
> > 10) ... well you get the idea.
> > Which, if any, would you write in Python?[/color]
>
> 6 and 8, maybe 3...and 10 of course[/color]

I think 6 and 8 would depend on decimal arithmetic; some modules for
that have been proposed but nothing is released yet, afaik.

I can't see answering 3 without also answering 2.

[Peter Hansen]

Re: Python for large projects

Paul Rubin wrote:[color=blue]
> I keep hearing this, but I don't see any large (much less very large)
> applications that have been done in Python; Zope is medium sized.
>
> Suppose you wanted to write any of the following:
>
> 1) Optimizing C/C++ compiler, like GCC
> 2) Full featured web browser, like MSIE or Mozilla
> 3) Full featured office suite, like MS Office or Open Office or KDE
> 4) Avionics for the space shuttle
> 5) Internals of a large telephone/data switch
> 6) Tax processing software for the IRS
> 7) Operating system kernel (Linux: the next generation)
> 8) Accounting software for a big bank
> 9) Full featured database like Oracle or Postgres
> 10) ... well you get the idea.
>
> Which, if any, would you write in Python? By "write in Python" I mean
> the central framework and most of the code is written in Python,
> though you're allowed to use the C API as needed. Using Python to
> provide some functions around the edges (e.g. the operator UI for the
> phone switch) doesn't count. Note that these are all supposed to be
> used in production and not just as technology demos, so speed matters
> (nobody wants a compiler that's 10x slower than GCC. GCC is slow
> enough already).
>
> I think I'd spend some time at least considering Python in each of the
> above cases, but I'm not sure I could convincingly make it fly for any
> of them.[/color]

D'oh! You ask this just before some of us have to leave for PyCon?!
How cruel! ;-)

(Hmm... maybe a BOF for "serious s/w engineering with Python" or
something like that?)

-Peter

[Jay O'Connor]

Re: Python for large projects

Paul Rubin wrote:
[color=blue]
> Jay O'Connor <jayoconnor@ear thlink.net> writes:
>[color=green][color=darkred]
>>>1) Optimizing C/C++ compiler, like GCC
>>>2) Full featured web browser, like MSIE or Mozilla
>>>3) Full featured office suite, like MS Office or Open Office or KDE
>>>4) Avionics for the space shuttle
>>>5) Internals of a large telephone/data switch
>>>6) Tax processing software for the IRS
>>>7) Operating system kernel (Linux: the next generation)
>>>8) Accounting software for a big bank
>>>9) Full featured database like Oracle or Postgres
>>>10) ... well you get the idea.
>>>Which, if any, would you write in Python?[/color]
>>
>>6 and 8, maybe 3...and 10 of course[/color]
>
>
> I think 6 and 8 would depend on decimal arithmetic; some modules for
> that have been proposed but nothing is released yet, afaik.[/color]

Depends on what you mean. If you're talking desktop line-of-business software
for the IRS, a bank, or any business then there's probably not going to be a lot
of heavy math. If you're talking some sort of automatic transaction processing
then maybe different, but I can't see the math really being that hoary. I've
seen financial transaction software written in TCL which was running a *lot*
slower than Python. Even in that case, I/O for getting yuor transactions and
storing your results are going to be over head so I would really discount Python
unless the example could be defined better. I'd seriously consider Python to
start with (but I'd consider Smalltalk first so..)
[color=blue]
> I can't see answering 3 without also answering 2.[/color]

A browser is a fairly straightforward piece of work that I think any slowness in
a particular language is going to be noticed when you repeat a fundamental
operation (render an element) many, many times. I think there's going to be a
lot more 'stuff' going on in office suite. That's just speculation, the real
reason I didn't consider #2 was that I overlooked it

Take care,
Jay

[Ville Vainio]

Re: Python for large projects

>>>>> "Peter" == Peter Hansen <peter@engcorp. com> writes:

Peter> (Hmm... maybe a BOF for "serious s/w engineering with
Peter> Python" or something like that?)

We're living in interesting times, what with all the noise created by
Havoc Pennington in

http://ometer.com/desktop-language.html

I think Python might have a great shot as a more "official" strategic
direction for Linux desktop app development, since it's quite possible
their neither Mono or Java gets picked because they are encumbered.

Some lobbying is needed, of course. Getting something like IBM or
Novell (well, Novell is probably stuck w/ Mono... IronPython, anyone?)
officially endorsing and sponsoring Python would be oh-so-cool.

--
Ville Vainio http://tinyurl.com/2prnb

[Jacek Generowicz]

Re: Python for large projects

Andrew Wilkinson <ajw140NO@SPAMy ork.ac.uk> writes:
[color=blue]
> assaf__@walla.c om wrote:[/color]
[color=blue]
> I find the lack of static type checking to cause some bugs to hide
> in my code that in other languages would be found at compile time.[/color]

I feel honour-bound to point out that citing static typing (explicit
static typing, in particular) as a means of creating more correct
programs, to be one of the greatest contemporary myths of software
engineering.

Static typing makes it easier for compilers to produce more efficient
code. That is the only advantage that static typing offers.

I am of the opinion that (explicit) static typing contributes to the
bugginess of programs.

[Neil Hodgson]

Re: Python for large projects

Jacek Generowicz:[color=blue]
> Do you have a reference describing such an attempt?[/color]

It was mentioned in Dr Dobbs, July 1996, page 87 although I also heard
about this from a colleague that worked with someone ...
Googling for "zero defect development", looks like the Dr Dobbs piece
requires an account

http://www.ddj.com/documents/s=959/ddj9607j/

Neil