Is there a "Large Scale Python Software Design" ?

[Andreas Kostyrka]

Re: Is there a "Large Scale Python Software Design" ?

On Tue, Oct 19, 2004 at 07:16:01AM -0700, Jonathan Ellis wrote:[color=blue]
> Testing is good; preventing entire classes of errors from ever
> happening at all is better, particularly when you get large. Avoiding
> connectedness helps, but that's not always possible.[/color]
What classes of errors are completely avoided by "static typing" as
implemented by C++ (Java)? Just out of curiosity, because this is
usually stated as "true by axiomatic definition" in this kind of
discussions.

Andreas

[Dave Brueck]

Re: Is there a "Large Scale Python Software Design" ?

Josiah Carlson wrote:[color=blue][color=green]
>>Heh. "Large" depends on a lot of things, particularly connectedness,
>>but I really can't picture 10k being large under any circumstances.[/color]
>
>
> Ok, so what is large? How many orders of magnitude larger than 10k
> lines does it take for a piece of software to be large? And why should
> you be the judge?[/color]

I think the only way to compare projects is from a user's or customer's
perspective - what functionality the application provides & its scope. Any
comparison involving lines of code or number of developers won't be reliable
unless other factors (especially implementation language & libraries) are held
semi-constant. For example, at one company I think the total was 1.1 or 1.2
million lines of code (all C++ & about 60-70 developers), and yet I have trouble
imagining how, if I could go back and do it again in Python, it'd take even 200k
lines of code (and the riskier side of me feels it'd come in at under 100k - it
just didn't _do_ a lot despite all that code!)

In that sense, a 10k Python app can be fairly large in terms of end-user
functionality. For example, our main product where I work consists of *many*
different custom servers, a full web-based administrative interface, an end-user
web interface, a client application that does all sorts of interaction with the
servers, and lots of database interaction. Add to this many internal tools,
integration tools we provide to our customers, etc., and I would rate it overall
as on the upper end of medium-sized projects, functionality-wise - not the
largest I've worked on but well beyond any definition of small, and our plans
for the next few quarters will definitely push it into the range of what I'd
normally consider a large system. IIRC we're only in the 10k-20k for lines of
Python code, plus a few modules here and there being C++.

Having said all that, I've found that competitors in our same space tend to have
20-30 developers on the low end to over 100 on the high-end, while we have but a
handful. We don't have quite the same breadth of functionality - at least not
yet - but we generally make up for it by accounting for it architecturally but
not adding it until a customer actually needs it (a sort of JIT approach to
development). As such we've been able to compete head-to-head with others in the
same sector. On more than one occasion I've wondered aloud how so many
developers working for Competitor X can stay busy, and I can only imagine how
many lines of code they're churning out - and yet, from a functionality
perspective we're keeping pace. I also wonder how many hours a day they spend in
meetings trying to coordinate everything. Ugh.

Back to the point at hand: a project using a higher-level language gets out of
hand more slowly; if there were no other advantage it'd still be a "win" IMO
because you encounter "big project" problems a lot later - and that's a huge
benefit in and of itself.

[Dave Brueck]

Re: Is there a "Large Scale Python Software Design" ?

Andreas Kostyrka wrote:[color=blue]
> On Tue, Oct 19, 2004 at 07:16:01AM -0700, Jonathan Ellis wrote:
>[color=green]
>>Testing is good; preventing entire classes of errors from ever
>>happening at all is better, particularly when you get large. Avoiding
>>connectedne ss helps, but that's not always possible.[/color]
>
> What classes of errors are completely avoided by "static typing" as
> implemented by C++ (Java)?[/color]

I'm curious as well, because from what I've seen, the classes of errors "caught"
are (1) a subset of the higher-level (e.g. algorithmic and corner-case) errors
caught by good testing anyway, (2) much more common in code written by
lazy/underexperience d developers who are already considered a liability, and (3)
caused in part by complexities introduced by the language itself*.

More modern/advanced static type systems that let you actually get into the
semantics of the program (as opposed to just deciding which predefined type
bucket your data fits in) may help, but IMO the jury's still out on them (partly
due to complexity, and partly due to _when_ in the development process they must
be defined - perhaps that's the root problem of some static type systems - they
make you declare intent and semantics when you know the _least_ about them!
Consider the parallels to available knowledge in compile-time versus run-time
optimizations).

-Dave

* A trivial example:When programmers need to count something, rarely do they
care about unsigned vs signed or short vs normal vs long vs longlong, and yet in
something like C++ they are _constantly_ making this decision.

Another: in Java, every exception that can be thrown must be mentioned in the
code every step of the way - a maintenance nightmare, not to mention the utter
distraction during development.

[Alex Martelli]

Re: Is there a "Large Scale Python Software Design" ?

Dave Brueck <dave@pythonapo crypha.com> wrote:
[color=blue]
> Andreas Kostyrka wrote:[color=green]
> > On Tue, Oct 19, 2004 at 07:16:01AM -0700, Jonathan Ellis wrote:
> >[color=darkred]
> >>Testing is good; preventing entire classes of errors from ever
> >>happening at all is better, particularly when you get large. Avoiding
> >>connectedne ss helps, but that's not always possible.[/color]
> >
> > What classes of errors are completely avoided by "static typing" as
> > implemented by C++ (Java)?[/color][/color]

C++'s casting power makes this a bit moot -- I have seen generally-good
developers (not quite comfy with C++, from a mostly-Fortran then a
little C background) mangle poor innocent rvalues (and even lvalues,
BION, with ample supplies of & and * to help) with such overpowering
hits of reinterpret_cas t<> that I'm still queasy to think of it years
later. Java is mercifully a bit less powerful, but of course _its_
casts are generally runtime-checked. So, when one sees:

WhatAWonderfulW ord w = (WhatAWonderful Word) v;

one _IS_ admittedly inclined to think that the "class of error being
completely avoided" is "erroneous omission of a cast that plays no
useful role at all and is going to be checked only at runtime anyway".

However, there _are_ tiny but undeniable advantages to static typing:

1. some typos are caught at compiletime, rather than 2 seconds later by
unit tests -- 2 seconds ain't much, but it ain't 0 either;

2. simple-minded tools have an easier time offering such editing
services as "auto-completion", which may save a little typing;

3. simple-minded compilers have an easier time producing halfway
decent code;

and the like. None deal with "classes of errors completely avoided"
unless one thinks of unittests as an optional add-on and of compilers as
a mandatory must-have, which is wrong -- the point Robert Martin makes
excellently in his artima article about the wonders of dynamic typing of
a bit more than a year ago (dynamic typing is wonderful _with_ unit
testing, but then unit testing is an absolute must anyway, to
summarize).

[color=blue]
> I'm curious as well, because from what I've seen, the classes of errors
> "caught" are (1) a subset of the higher-level (e.g. algorithmic and
> corner-case) errors caught by good testing anyway,[/color]

Yes, undeniable.
[color=blue]
> (2) much more common in code written by
> lazy/underexperience d developers who are already considered a liability,[/color]

No, I think you're wrong here. Typos are just as frequent for just
about all classes of coders, lazy or eager, experienced or not -- the
eager experienced ones often use faster typing (nothing to do with
static typing;-).
[color=blue]
> and (3)
> caused in part by complexities introduced by the language itself*.[/color]

Yes, a fair cop. E.g., a typo in one of those redundant mentions of a
type or interface, seen above, is an error introduced only because I'm
required to type the GD thing twice over (though autocompletion may save
me some keystrokes;-).

[color=blue]
> More modern/advanced static type systems that let you actually get into
> the semantics of the program (as opposed to just deciding which predefined
> type bucket your data fits in) may help, but IMO the jury's still out on
> them (partly due to complexity, and partly due to _when_ in the
> development process they must be defined - perhaps that's the root problem
> of some static type systems - they make you declare intent and semantics
> when you know the _least_ about them! Consider the parallels to available
> knowledge in compile-time versus run-time optimizations).[/color]

If you mean typesystems such as Haskell's or ML's, allowing extended
inference (and, in Haskell's case, the wonder of typeclasses), I think
you're being a bit unfair here. You can refactor your types and
typeclasses just as much as any other part of your code, so the "when
they must be defined" seems a bit of a red herring to me (unless you
have in mind other more advanced typesystems yet, in which case I'd like
some URL to read up on them -- TIA).

I think we agree at 95% to 99%, btw, I admit I'm just picking nits...


Alex

[Alex Martelli]

Re: Is there a &quot;Large Scale Python Software Design&quot; ?

Josiah Carlson <jcarlson@uci.e du> wrote:
[color=blue][color=green]
> > Heh. "Large" depends on a lot of things, particularly connectedness,
> > but I really can't picture 10k being large under any circumstances.[/color]
>
> Ok, so what is large? How many orders of magnitude larger than 10k
> lines does it take for a piece of software to be large? And why should
> you be the judge?[/color]

My definition of a large software system is: a system that cannot
sensibly be developed and maintained by just one developer, but requires
a team of developers. Among the factors defining where the boundaries
lie are such things as deployment issues (how many platforms, how
diverse), function points, analysis/requirements, etc, etc, but SLOC
(properly counted/normalized lines of code) are the main determinant.

For a reasonably experienced programmer, with decent tools, and without
hair-raising problems of deployment, optimization, continuous fast
changes to specs, etc, etc, 10k SLOC should be within the threshold of
"can be sensibly developed and maintained by one person"; 100k SLOC
won't be; the threshold is somewhere in-between. Of course, if you're
talking freshman programming trainees, or special problems of the
various sorts mentioned, the thresholds do shift downwards.

[color=blue]
> I'd let it slip to medium, but I wouldn't say that the project was small.
> Small is something you can do in a weekend because you've been putting
> it off. Small is something a newb to the language can do in a week
> while they are learning the language.[/color]

OK, that's your definition of "small", I guess. I don't know that
there's a commonly accepted one. On the other hand, moving from a
project that can all fit in your head, one you can fully develop and
actively maintain by yourself, to a team situation, _is_ a crucial
threshold, as teams have such different strengths and problems than
individuals on their own; and the "Large Scale" monicker is typically
tagged onto projects requiring a team.

We can quibble about special cases (is a 2-people team, with one of them
developing half-time and the rest of the time out selling the system,
comparable to a more typical case of 6-10 people working full-time on
development and maintenance of a system?), but that's always so for
taxonomies, and doesn't add much to the discussion IMHO.


Alex

[Greg Ewing]

Re: Is there a &quot;Large Scale Python Software Design&quot; ?

Andrew Dalke wrote:[color=blue]
> I didn't know what was going on at a certain
> spot, looked up a few lines, and saw the comment I had
> written explaining the tricky spot. I thought it was
> very nice of the past me to help out then present me. :)[/color]

Indeed. It would be nice to have access to a time machine
so one could go back and ask oneself about things like
this...

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand

http://www.cosc.canterbury.ac.nz/~greg

[Andrew Dalke]

Re: Is there a &quot;Large Scale Python Software Design&quot; ?

Greg Ewing wrote:[color=blue]
> Indeed. It would be nice to have access to a time machine
> so one could go back and ask oneself about things like
> this...[/color]

As in James P. Hogan's book "Thrice Upon a Time".


Also notable for having no antagonist and for being
the only story I know of that uses quantum time as
the way to resolve time travel paradoxes.

Andrew
dalke@dalkescie ntific.com

[Michele Simionato]

Re: Is there a &quot;Large Scale Python Software Design&quot; ?

aleaxit@yahoo.c om (Alex Martelli) wrote in message news:<1glwwg1.1 hgx2561vflstbN% aleaxit@yahoo.c om>...[color=blue]
> That is the book I want to write, the one I have always wanted to write;
> the Nutshell and the Cookbook (and now their second editions) keep
> delaying that plan, but, in a sense, that's good, because I keep
> accumulating useful experiences to enrich those notes, and Python keeps
> growing (particularly but not exclusively in terms of third-party
> extensions and tools) in ways that refine and sometimes indeed redefine
> some key aspects. To give a simple technical example: I used to have
> substantial caveats in those notes cautioning readers to use multiple
> inheritance in an extremely sparing, cautious way, due to traps and
> pitfalls that made it fragile. Nowadays, with the advent of 2.3, most
> of those traps and pitfalls have gone away (in the newstyle object
> model), to the point that the whole issue can be reconsidered.
>[/color]

Uhm ... I must say that I was quite keen of Multiple Inheritance
but having seen the (ab)use of it in Zope I am starting questioning
the wisdom of it. The problem I see with MI (even done well) is that
you keep getting methods from parent classes and each time you have
to think about the MRO and the precedence rules. It is an additional
burden in the programmer's mind. I miss the clean simple concept of
superclass; the MRO may be cool but it is not as simple to learn, to
teach
and especially remember. Notice, I am not referring to the algorithm,
it is
not important to remember it; what is disturbing to me is to be aware
that the resolution of the methods can be non-trivial and that I
should call .mro() each time to check exactly what is happening.
Also 'super' is hard to understand and to use :-(
So, I wonder if Matz was right after all and single inheritance +
mixins
à la Ruby are the right way to go. Yes, from a purist point of view
they are inferior to MI, however from the pragramatist point
of view I don't think you loose very much, and you get a big gain in
short learning curve and expliciteness. Especially 'super' stays
simple.

However I lack experience in Ruby with mixins: do you have experience
or do you know people with experience on that? What they think?
Are they happy with the approach or they wish Ruby had real MI?
Of course in simple systems there is no real issue, I am talking
about large systems. Also, I am not talking about wrong design choice
(for
instance Zope 3 use MI much less than Zope 2: I interpret this as a
recognition that the design was wrong) but in general: assuming you
have
an application where the "right" design is via mixins, is there a real
difference in doing it à la Ruby or with real MI?
It does not look there is a big difference, in practice.
Yes, you do not have the full power of cooperative methods but you
also avoid the burden of them and you can always find workarounds; I
would
say there are compensations.

I have not yet a definite opionion on this point, so I would like to
hear the opinion of others, especially people with real world
experience in complex
systems.

Michele Simionato

[Jonathan Ellis]

Re: Is there a &quot;Large Scale Python Software Design&quot; ?

Stephen Waterbury wrote:[color=blue]
> Jonathan Ellis wrote:
>[color=green]
> > ... A more-experienced co-worker pointed me in the right
> > direction, and the IDE did the rest. ("Find definition," "Find
> > references.") Grep can do much the same thing, but painfully[/color][/color]
slowly --[color=blue][color=green]
> > and inaccurately, when you have a bunch of interfaces implementing[/color][/color]
the[color=blue][color=green]
> > same method names. ...[/color]
>
> Try "glimpse" (http://webglimpse.net) -- it uses a superset of
> grep's arguments and can search large collections of files at
> a single bound! Re-indexing takes a few seconds, but doesn't
> need to be done unless there are major changes.[/color]

glimpse addresses grep's speed problem, but unfortunately has no more
semantic understanding beyond "it's just text." Etags is a little
better but not much, and also suffers from the
have-to-remember-to-reindex-if-you-want-accurate-results "feature."
-Jonathan

[Jonathan Ellis]

Re: Is there a &quot;Large Scale Python Software Design&quot; ?

Andreas Kostyrka wrote:[color=blue]
> On Tue, Oct 19, 2004 at 07:16:01AM -0700, Jonathan Ellis wrote:[color=green]
> > Testing is good; preventing entire classes of errors from ever
> > happening at all is better, particularly when you get large.[/color][/color]
Avoiding[color=blue][color=green]
> > connectedness helps, but that's not always possible.[/color][/color]
[color=blue]
> What classes of errors are completely avoided by "static typing" as
> implemented by C++ (Java)? Just out of curiosity, because this is
> usually stated as "true by axiomatic definition" in this kind of
> discussions.[/color]

As one example: in this codebase (closer to 700 kloc than 500 by this
time, if it matters) the very oldest code used a Borland wrapper over
JDBC. At the time, it allowed doing things JDBC version 1 did not; by
the time I got fed up, JDBC version 3 had caught up and far surpassed
Borland's API. There was also a lot of JDBC code that was suboptimal
-- for the application I worked on, it almost always made sense to use
a PreparedStateme nt rather than a simple Statement, but because binding
parameters in jdbc is something of a PITA we often went with the
Statement anyway. Both the Borland-style and the JDBC code also dealt
with calls to stored procedures, most of them not in CallableStateme nts
(the "right" way to do this).

I volunteered to write a more friendly wrapper over JDBC than Borland's
that would handle caching of [Prepared|Callab le]Statement objects and
parameter binding transparently, nothing fancy (in particular my select
methods returned ResultSets, where Borland had their own class for
this) and rewrite these thousands of calls to use the new API. Of
course I wrote scripts to do this; 5 or 6, each handling a different
aspect.

To write unit tests for this by hand would have been obscene. (As an
aside, writing unit tests for anything that deals with many tables in a
database is a PITA already and usually ends up not really a "unit" test
anymore.) Even generating unit tests with more scripts would have
required a significantly deeper semantic understanding of the code
being filtered, and hence a lot more work.

As it was, with the compiler letting me know when I screwed up and
improve my scripts accordlingly, out of the thousands of calls, I
ultimately had to do a few dozen by hand (because that was less work
than getting my scripts able to deal with the very worst examples), and
the compiler let me know what those were. After the process was
complete, QA turned up (over several weeks) 4 or 5 places where I'd
broken things despite the static checking, which I considered a very
good success ratio.

-Jonathan

[Jonathan Ellis]

Re: Is there a &quot;Large Scale Python Software Design&quot; ?

Peter Hansen wrote:[color=blue]
> Jonathan Ellis wrote:[color=green]
> > I haven't jumped into a project of similar size with python, but[/color][/color]
the[color=blue][color=green]
> > tool support for this approach to working with a large codebase[/color][/color]
just[color=blue][color=green]
> > isn't there, and I haven't seen any convincing arguments that
> > alternative methodologies are enough better to make up for this.[/color]
>
> I'm getting the impression you also haven't tried any significant
> test-driven development. The benefits of this approach are *easily*
> convincing to most people, and it also fits the bill as removing the
> need for a very sizable portion of the tool support which you rightly
> point out is not there in most tools for dynamically typed languages.[/color]

I think I responded to this already --
[color=blue][color=green]
> > Testing is good; preventing entire classes of errors from ever
> > happening at all is better, particularly when you get large.[/color][/color]
Avoiding[color=blue][color=green]
> > connectedness helps, but that's not always possible.[/color][/color]

Oh yes; so I did. :) (See my reply to another subthread for one
example of when static type checking saved me a LOT of work.)

What is the biggest system you have built with python personally? I'm
happy to be proven wrong, but honestly, the most enthusiastic "testing
solves all my problem" people I have seen haven't worked on anything
"large" -- and my definition of large agrees with Alex's; over 100
kloc, more than a handful of developers.

So people don't get me wrong: I love python. Most of my programming
friends call me "the python zealot" behind my back. I just don't think
it's the right tool for every problem.

Specifically, in my experience, statically-typed languages make it much
easier to say "okay, I'm fixing a bug in Class.Foo; here's all the
places where it's used." This lets me see how Foo is actually used --
in a perfect world, Foo's documentation is precise and up to date, but
I haven't worked anywhere that this was always the case -- which lets
me make my fix with a reasonable chance of not breaking anything.
Compile-time type checking increases those chances. Unit tests
increase that further, but relying on unit tests as your first and only
line of defense is suboptimal when there are better options.
[color=blue]
> Having experience with both approaches, and choosing one over
> the other, gives one greater credibility than having experience
> with just one approach, yet clinging to it...[/color]

You are incorrect if you assume I am unfamiliar with python. I readily
admit I have no experience with truly large python projects; I would
classify my the python application I work on as "small," but it seems I
am in good company here in that respect... I do claim to have fairly
extensive experience with large projects in a statically typed language
(Java).

-Jonathan

[Alex Martelli]

Re: Is there a &quot;Large Scale Python Software Design&quot; ?

Jonathan Ellis <jbellis@gmail. com> wrote:
[color=blue]
> What is the biggest system you have built with python personally? I'm
> happy to be proven wrong, but honestly, the most enthusiastic "testing
> solves all my problem" people I have seen haven't worked on anything
> "large" -- and my definition of large agrees with Alex's; over 100
> kloc, more than a handful of developers.[/color]

I have the experience, both with Python and with C++, and I can confirm
that test-driven development (with more code for tests, particularly
unit- but also system-/integration-/acceptance-, than code to implement
actual functionality) scales up.

The C++ system had about five times the number of developers and ten
times the code size for about the same amount of functionality (as
roughly measured in function points) as the Python system.

Type safety and const-correctness in the C++ system were of very minor
help; not 100% negligible, but clearly they were not pulling their
weight, by a long shot.

In both systems, the trouble spots came invariably where testing had
been skimped on, due to time pressures and insufficient acculturation of
developers to testing; the temptation to shirk is a bit bigger in C++,
where one can work under the delusion that the compiler's typechecks
compensate (they don't).

_Retrofitting_ tests to code developed any old how is not as effective
as growing the tests and code together. It appears to me that the
experience you relate is about code which didn't have a good battery of
unit tests to go with it.

Lastly, I'm still looking for systematic ways to test system integration
that are as effective as unit tests are for each single component or
subsystem; but that's an area where type and const checking are of just
about negligible help.


Alex

[GerritM]

Re: Is there a &quot;Large Scale Python Software Design&quot; ?

"Alex Martelli" <aleaxit@yahoo. com> schreef in bericht
news:1gm1x7t.w8 k2hqrv300tN%ale axit@yahoo.com. ..
<...skip...>[color=blue]
> Lastly, I'm still looking for systematic ways to test system integration
> that are as effective as unit tests are for each single component or
> subsystem; but that's an area where type and const checking are of just
> about negligible help.
>[/color]
System integration has a completely different nature than unit testing.
During system integration the "unforeseen s" and the "unknowns" pop-up. And
of course the not-communicated, implicit human assumptions are uncovered.
And the "non-functional" behavior is a source of problems (response times,
memory footprint, etc). Many system integration problems are semantic
problems. system integration is often dufficult due to heterogeniety of the
problems, technologies and the people involved. In other words the larger
the system the more challenging systems integration becomes

All of these problems are not addressed at all by static typing. However,
design clarity and compactness does help tremendously. I would expect for
these reasons that Python is a big plus during system integration of large
systems. Of course design attention is required to cope with the
"non-functional" imapct of Python, such as CPU and memory consumption. on
top of that (run-time) instrumentation is very helpful. Here again the
dynamic nature of Python is a big plus.

kind regards, Gerrit Muller
Gaudi Systems Architecting www.extra.research.philips.com/natlab/sysarch/

[Alex Martelli]

Re: Is there a &quot;Large Scale Python Software Design&quot; ?

GerritM <gmuller@worldo nline.nl> wrote:
[color=blue]
> "Alex Martelli" <aleaxit@yahoo. com> schreef in bericht
> news:1gm1x7t.w8 k2hqrv300tN%ale axit@yahoo.com. ..
> <...skip...>[color=green]
> > Lastly, I'm still looking for systematic ways to test system integration
> > that are as effective as unit tests are for each single component or
> > subsystem; but that's an area where type and const checking are of just
> > about negligible help.
> >[/color]
> System integration has a completely different nature than unit testing.
> During system integration the "unforeseen s" and the "unknowns" pop-up. And
> of course the not-communicated, implicit human assumptions are uncovered.[/color]

Exactly -- which is why I'm still looking (doesn't mean I think I'll
find;-).
[color=blue]
> All of these problems are not addressed at all by static typing. However,[/color]

Essentially not.
[color=blue]
> design clarity and compactness does help tremendously. I would expect for
> these reasons that Python is a big plus during system integration of large[/color]

Not as much as one might hope, in my experience. Protocol Adaptation
_would_ help (see PEP 246), but it would need to be widely deployed.
[color=blue]
> systems. Of course design attention is required to cope with the
> "non-functional" imapct of Python, such as CPU and memory consumption. on
> top of that (run-time) instrumentation is very helpful. Here again the
> dynamic nature of Python is a big plus.[/color]

But the extreme difficulty in keeping track of what amount of memory
goes where in what cases is a big minus. I recall similar problems with
Java, in my limited experience with it, but for Java I see now there are
commercial tools specifically to hunt down memory problems. In C++
there were actual _leaks_ which were a terrible problem for us, but
again pricey commercial technology came to the rescue.

With Python, I've found, so far, that tracking where _time_ goes is
quite feasible, with systematic profiling &c (of course profiling is
always a bit invasive, and so on, but no more so in Python than
otherwise), so that in the end CPU consumption is no big deal (it's easy
to find out the tiny hot spot and turn it into an extension iff needed).
But memory is a _big_ problem, in my experience so far, with servers
meant to run a long time and having very large code bases. I'm sure
there IS a commercial niche for a _good_ general purpose Python tool to
keep track of memory consumption, equivalent to those available for C,
C++ and Java...


Alex

[Peter Hansen]

Re: Is there a &quot;Large Scale Python Software Design&quot; ?

Jonathan Ellis wrote:[color=blue]
> Peter Hansen wrote:[color=green]
>>I'm getting the impression you also haven't tried any significant
>>test-driven development.[/color]
>
> I think I responded to this already --
>[color=green][color=darkred]
>>>Testing is good; preventing entire classes of errors from ever
>>>happening at all is better, particularly when you get large.[/color][/color]
>
> Oh yes; so I did. :) (See my reply to another subthread for one
> example of when static type checking saved me a LOT of work.)[/color]

And you've reemphasized my point. "Testing" is not test-driven
development. In fact, test-driven development is about *design*,
not just about testing. The two are related, but definitely not
the same thing, and eliminating TDD with a wave of a hand intended
to poo-poo mere testing is to miss the point. Once someone has
tried TDD, they are unlikely to lump it in with simple "unit testing"
as it has other properties that aren't obvious on the surface.
[color=blue]
> What is the biggest system you have built with python personally? I'm
> happy to be proven wrong, but honestly, the most enthusiastic "testing
> solves all my problem" people I have seen haven't worked on anything
> "large" -- and my definition of large agrees with Alex's; over 100
> kloc, more than a handful of developers.[/color]

The topic of the thread was large projects with _large teams_,
I thought, so I won't focus on my personal work. The team I
was leading worked on code that, if I recall, was somewhat over
100,000 lines of Python code including tests. I don't recall
whether that number was the largest piece, or combining several
separate applications which ran together but in a distributed
system... I think there were close to 20 man years in the main
bit.

(And remembering that 1 line of Python code corresponds to
some larger number, maybe five or ten, of C code, that should
qualify it as a large project by many definitions.)
[color=blue]
> So people don't get me wrong: I love python. Most of my programming
> friends call me "the python zealot" behind my back. I just don't think
> it's the right tool for every problem.[/color]

Neither do I. The above project also involved some C and
some assembly, plus some Javascript and possibly something else
I've forgotten by now. We just made efforts to use Python *as
much as possible* and it paid off.
[color=blue]
> Specifically, in my experience, statically-typed languages make it much
> easier to say "okay, I'm fixing a bug in Class.Foo; here's all the
> places where it's used." This lets me see how Foo is actually used --
> in a perfect world, Foo's documentation is precise and up to date, but
> I haven't worked anywhere that this was always the case -- which lets
> me make my fix with a reasonable chance of not breaking anything.
> Compile-time type checking increases those chances. Unit tests
> increase that further, but relying on unit tests as your first and only
> line of defense is suboptimal when there are better options.[/color]

But what if you already had tests which allowed you to do exactly
the thing you describe? Is there a need for "better options"
at that point? Are they really better? When I do TDD, I can
*trivially* catch all the cases where Class.Foo is used
because they are all exercised by the tests. Furthermore, I
can catch real bugs, not just typos and simple things involving
using the wrong type. A superset of the bugs your statically
typed language tools are letting you catch. But obviously
I'm rehashing the argument, and one which has been discussed
here many times, so I should let it go.
[color=blue][color=green]
>>Having experience with both approaches, and choosing one over
>>the other, gives one greater credibility than having experience
>>with just one approach, yet clinging to it...[/color]
>
> You are incorrect if you assume I am unfamiliar with python.[/color]

I assumed no such thing, just that you were unfamiliar with
large projects in Python and yet were advising the OP on its
suitability in that realm. You're bright and experienced, and
your comments have substance, but until you've actually
participated in a large project with Python and seen it fail
gloriously *because it was not statically typed*, I wouldn't
put much weight on your comments in this area if I were the
OP. That's all I was saying...

-Peter