Is there a "Large Scale Python Software Design" ?

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Andrea Griffini

    #1

    Is there a "Large Scale Python Software Design" ?


    I did it.

    I proposed python as the main language for our next CAD/CAM
    software because I think that it has all the potential needed
    for it. I'm not sure yet if the decision will get through, but
    something I'll need in this case is some experience-based set
    of rules about how to use python in this context.

    For example... is defining readonly attributes in classes
    worth the hassle ? Does duck-typing scale well in complex
    software or should I go for a classic inheritance hierarchy ?

    In other words... is there something like the classic "Large
    Scale C++ Software Design" (Lakos) for python ? I'm not
    looking for a bible, but lessons learned from someone that
    already went down this path could be quite interesting.

    Any suggestions/pointers are welcome.

    Andrea
  • Jonathan  Ellis

    #2
    Re: Is there a "Large Scale Python Software Design" ?

    Andrea Griffini wrote:[color=blue]
    > I did it.
    >
    > I proposed python as the main language for our next CAD/CAM
    > software because I think that it has all the potential needed
    > for it. I'm not sure yet if the decision will get through, but
    > something I'll need in this case is some experience-based set
    > of rules about how to use python in this context.
    >
    > For example... is defining readonly attributes in classes
    > worth the hassle ? Does duck-typing scale well in complex
    > software or should I go for a classic inheritance hierarchy ?
    >
    > In other words... is there something like the classic "Large
    > Scale C++ Software Design" (Lakos) for python ? I'm not
    > looking for a bible, but lessons learned from someone that
    > already went down this path could be quite interesting.[/color]

    Wouldn't it have been better to ask these questions BEFORE proposing
    python as (presumably) a Great Solution? IMO, as great as python is,
    it isn't appropriate for projects that are large and include many
    developers.

    The benefits of static typing, not least among which is the vastly
    superior ease of creating tools that "understand " the language,
    outweigh python's advantages in an environment when many people are
    writing a lot of code. This can be mitigated by reducing the
    connectedness of your code, e.g. with a plugin architecture, but that
    isn't always an option either...

    Good luck.

    -Jonathan

    Comment

    • Carlos Ribeiro

      #3
      Re: Is there a "Large Scale Python Software Design" ?

      On 18 Oct 2004 16:49:29 -0700, Jonathan Ellis <jbellis@gmail. com> wrote:[color=blue]
      > The benefits of static typing, not least among which is the vastly
      > superior ease of creating tools that "understand " the language,
      > outweigh python's advantages in an environment when many people are
      > writing a lot of code. This can be mitigated by reducing the
      > connectedness of your code, e.g. with a plugin architecture, but that
      > isn't always an option either...[/color]

      On principle, I disagree with this statement. Doing large scale
      development using Python isn't certainly the same thing as to do it
      with another language - C, C++ or Java, for instance. It will require
      a different approach to the problem, and perhaps a particular set of
      tools and disciplines to help with the process. But I don't think that
      static typing represents such a great advantage per itself, as to make
      Python badly suited to the problem, because there are many aspects to
      it, and Python has its own advantages too. Give it a solid design,
      leveraging Python particular strengths, and the end result has the
      potential be a positive surprise. But again, that's just my opinion,
      and I'm not the best person around to make a definitive claim on it
      :-)


      --
      Carlos Ribeiro
      Consultoria em Projetos
      blog: http://rascunhosrotos.blogspot.com
      blog: http://pythonnotes.blogspot.com
      mail: carribeiro@gmai l.com
      mail: carribeiro@yaho o.com

      Comment

      • Peter L Hansen

        #4
        Re: Is there a &quot;Large Scale Python Software Design&quot; ?

        Jonathan Ellis wrote:[color=blue]
        > Andrea Griffini wrote:[color=green]
        >>I proposed python as the main language for our next CAD/CAM
        >>software because I think that it has all the potential needed
        >>for it.[/color]
        >
        > Wouldn't it have been better to ask these questions BEFORE proposing
        > python as (presumably) a Great Solution? IMO, as great as python is,
        > it isn't appropriate for projects that are large and include many
        > developers.[/color]

        I don't know what Jonathan's experience with using Python in
        large teams and projects is, but mine includes four years
        as Director of Software Engineering at a wireless tech company
        and a team that ran between ten or fifteen people, and a very
        large amount of code. We found Python to be *very* appropriate
        for this and of course anything smaller.
        [color=blue]
        > The benefits of static typing, not least among which is the vastly
        > superior ease of creating tools that "understand " the language,
        > outweigh python's advantages in an environment when many people are
        > writing a lot of code.[/color]

        While it appears true that it is easier to develop certain
        tools for statically typed languages, it's not at all apparent
        that this small benefit outweighs the very significant advantages
        that Python brings to large-scale development, and to large-team
        development. I'll add "especially when using test-driven
        development and any agile process", and to be perfectly honest
        I'm not sure I would recommend Python nearly as strongly if one
        was forced to use a traditional, non-agile approach to the work.

        My past posts on the subject have covered this a number of times.
        I have to admit I haven't seen anything from Jonathan on this
        topic, so I can't say how his experience compares with mine, nor
        why he would feel the way he does.

        -Peter

        Comment

        • Josiah Carlson

          #5
          Re: Is there a &quot;Large Scale Python Software Design&quot; ?

          Andrea Griffini wrote:[color=blue]
          > I proposed python as the main language for our next CAD/CAM
          > software because I think that it has all the potential needed
          > for it. I'm not sure yet if the decision will get through, but
          > something I'll need in this case is some experience-based set
          > of rules about how to use python in this context.[/color]

          I know of 2 startups who have decided to construct similar softwares in
          Python, due to the fact that they can build entire packages in a year
          with a small, but experienced, development team. At least one
          of them is funded in the tens-of-millions of dollars range by a
          half-dozen automotive and aerospace companies.


          Jonathan Ellis wrote:[color=blue]
          > Wouldn't it have been better to ask these questions BEFORE proposing
          > python as (presumably) a Great Solution? IMO, as great as python is,
          > it isn't appropriate for projects that are large and include many
          > developers.[/color]

          Having recently released a piece of software with 10k lines of Python
          running in its backend as a core technology, and being paid for it, I
          will say that Python was and is the best tool for the job. A C version
          would have been at least 4-10 times as many lines, and we wouldn't be
          releasing ~3 months after starting with nearly the confidence we are now.


          In terms of developers, some projects require more than one developer,
          and in that sense, Python works as well as other languages: planning is
          key.

          - Josiah

          Comment

          • Dave Brueck

            #6
            Re: Is there a &quot;Large Scale Python Software Design&quot; ?

            Jonathan Ellis wrote:[color=blue]
            > Andrea Griffini wrote:
            >[color=green]
            >>I did it.
            >>
            >>I proposed python as the main language for our next CAD/CAM
            >>software because I think that it has all the potential needed
            >>for it. I'm not sure yet if the decision will get through, but
            >>something I'll need in this case is some experience-based set
            >>of rules about how to use python in this context.
            >>
            >>For example... is defining readonly attributes in classes
            >>worth the hassle ? Does duck-typing scale well in complex
            >>software or should I go for a classic inheritance hierarchy ?
            >>
            >>In other words... is there something like the classic "Large
            >>Scale C++ Software Design" (Lakos) for python ? I'm not
            >>looking for a bible, but lessons learned from someone that
            >>already went down this path could be quite interesting.[/color]
            >
            >
            > Wouldn't it have been better to ask these questions BEFORE proposing
            > python as (presumably) a Great Solution? IMO, as great as python is,
            > it isn't appropriate for projects that are large and include many
            > developers.[/color]

            The OP would be well-advised to search the Google archives of c.l.py as many
            (myself included) take the contrarian view - as the project grows in size it is
            harder to justify going with "classic" languages like C++, or even Java - the
            associated costs at each stage of the project are relatively larger to begin
            with, and grow more quickly as well.
            [color=blue]
            > The benefits of static typing, not least among which is the vastly
            > superior ease of creating tools that "understand " the language,
            > outweigh python's advantages in an environment when many people are
            > writing a lot of code.[/color]

            I'm not so sure - how much of the benefit those "smart" tools provide goes to
            helping the developer manage complexity caused by the language itself? It seems
            that often (not always, of course) a lot of what they do is help the programmer
            manage oodles of little details that the programmer ought not be burdened with
            in the first place, _especially_ on large projects.

            What specifically do you see breaking down if Python is used in a project with
            lots of people? From working on large projects with lots of people, I've noticed
            that projects naturally get divided into components as different teams work on
            them, regardless of the language (so for any given piece of code, the percentage
            of the total programmers touching that piece of code drops, not rises, as the
            total size of the development staff goes up). Again, regardless of language,
            large projects & teams almost force well-defined interface points between
            various components - I don't see how Python would be any hinderance at all.

            On the plus side, projects implemented in higher level languages grow more
            slowly (and thus become unmanageable more slowly) than would projects
            implemented in lower-level languages. The list goes on and on - I've found
            Python components generally easier to test than, say, C++ components. It's also
            easier for more people to comprehend more of the code (and, in turn, more of the
            implications of decisions), etc., etc.

            -Dave

            Comment

            • Stephen Waterbury

              #7
              Re: Is there a &quot;Large Scale Python Software Design&quot; ?

              Andrea Griffini wrote:
              [color=blue]
              > I proposed python as the main language for our next CAD/CAM
              > software because I think that it has all the potential needed
              > for it.[/color]

              I agree, even without knowing the intended scope. ;)
              Speaking of scope, if you are allowed to divulge it, that
              would be interesting to know. Will it be 2D or 3D (3D I
              would assume), and what kind of geometry engine?
              Probably one of the open-source ones that already have
              a Python API, no?

              If it is 3D, a very desirable feature would be STEP
              (ISO 10303) geometry import/export, so that you will be able
              to exchange CAD data with virtually any commercial CAD
              tool, and some open source ones (such as OpenCascade).
              That will greatly increase its chance of adoption by
              experienced CAD users, who typically have existing
              libraries of CAD designs created using a COTS CAD tool.
              (This is even more useful if you are planning to support
              assemblies of components -- which might even be the most
              logical initial feature for a new Python-based CAD/CAM,
              since assemblies could be manipulated even without having
              native geometric-form-creation capabilities: all you
              would need is rendering, orientation, and interfacing
              of existing solids -- a.k.a., "parts".)

              If you have access to a license for ABAQUS, I recently
              discovered that they have implemented a Python API for their
              FEA engine, and have implemented STEP geometry as well.
              See: http://www.abaqus.com/PAPortal
              [color=blue]
              > ... I'm not sure yet if the decision will get through, but
              > something I'll need in this case is some experience-based set
              > of rules about how to use python in this context.
              >
              > For example... is defining readonly attributes in classes
              > worth the hassle ? Does duck-typing scale well in complex
              > software or should I go for a classic inheritance hierarchy ?[/color]

              For something as complex as CAD/CAM, you will probably want to
              make maximum use of interfaces and adaptors, with minimal and
              very judicious application of classic inheritance hierarchies.
              I am *not* an expert on interfaces and adapters, but several
              of the gurus on this list are.

              Since you will probably want to do lots of prototyping, you
              can probably delay decisions about matters such as read-only
              attributes until your API has stabilized somewhat.

              Keep us posted on your progress.

              Cheers,
              Steve

              Comment

              • Jonathan  Ellis

                #8
                Re: Is there a &quot;Large Scale Python Software Design&quot; ?

                Josiah Carlson wrote:[color=blue]
                > Jonathan Ellis wrote:[color=green]
                > > Wouldn't it have been better to ask these questions BEFORE[/color][/color]
                proposing[color=blue][color=green]
                > > python as (presumably) a Great Solution? IMO, as great as python[/color][/color]
                is,[color=blue][color=green]
                > > it isn't appropriate for projects that are large and include many
                > > developers.[/color]
                >
                > Having recently released a piece of software with 10k lines of Python
                > running in its backend as a core technology, and being paid for it, I
                > will say that Python was and is the best tool for the job. A C[/color]
                version[color=blue]
                > would have been at least 4-10 times as many lines, and we wouldn't be
                > releasing ~3 months after starting with nearly the confidence we are[/color]
                now.

                Heh. "Large" depends on a lot of things, particularly connectedness,
                but I really can't picture 10k being large under any circumstances.
                -Jonathan

                Comment

                • Jonathan  Ellis

                  #9
                  Re: Is there a &quot;Large Scale Python Software Design&quot; ?

                  Peter L Hansen wrote:[color=blue]
                  > Jonathan Ellis wrote:[color=green]
                  > > The benefits of static typing, not least among which is the vastly
                  > > superior ease of creating tools that "understand " the language,
                  > > outweigh python's advantages in an environment when many people are
                  > > writing a lot of code.[/color]
                  >
                  > While it appears true that it is easier to develop certain
                  > tools for statically typed languages, it's not at all apparent
                  > that this small benefit outweighs the very significant advantages
                  > that Python brings to large-scale development, and to large-team
                  > development.[/color]

                  Almost four years ago I started working at a company with about 500
                  kloc of Java code. Thanks largely to tool support I was able to get in
                  and start fixing bugs my first day (this is without significant prior
                  Java experience). A more-experienced co-worker pointed me in the right
                  direction, and the IDE did the rest. ("Find definition," "Find
                  references.") Grep can do much the same thing, but painfully slowly --
                  and inaccurately, when you have a bunch of interfaces implementing the
                  same method names. Even after years in the codebase, I still used
                  these heavily; the codebase grew to about 800 kloc during the 3 years I
                  worked there. Developers came and went; even if my memory were good
                  enough to remember all the code _I_ ever wrote, I'd still have to
                  periodically repeat the familiarization process with code written by
                  others.

                  I haven't jumped into a project of similar size with python, but the
                  tool support for this approach to working with a large codebase just
                  isn't there, and I haven't seen any convincing arguments that
                  alternative methodologies are enough better to make up for this.
                  [color=blue]
                  > I'll add "especially when using test-driven
                  > development and any agile process", and to be perfectly honest
                  > I'm not sure I would recommend Python nearly as strongly if one
                  > was forced to use a traditional, non-agile approach to the work.[/color]

                  Testing is good; preventing entire classes of errors from ever
                  happening at all is better, particularly when you get large. Avoiding
                  connectedness helps, but that's not always possible.

                  -Jonathan

                  Comment

                  • Brad Tilley

                    #10
                    Re: Is there a &quot;Large Scale Python Software Design&quot; ?

                    Jonathan Ellis wrote:[color=blue]
                    > Heh. "Large" depends on a lot of things, particularly connectedness,
                    > but I really can't picture 10k being large under any circumstances.
                    > -Jonathan
                    >[/color]

                    It's large to me. Most sys-admin scripts/programs never cross 1K or 2K
                    at the most. And Python works well for sys-admin tasks.

                    Comment

                    • Alex Martelli

                      #11
                      Re: Is there a &quot;Large Scale Python Software Design&quot; ?

                      Dave Brueck <dave@pythonapo crypha.com> wrote:
                      [color=blue]
                      > The OP would be well-advised to search the Google archives of c.l.py as
                      > many (myself included) take the contrarian view - as the project grows in
                      > size it is harder to justify going with "classic" languages like C++, or
                      > even Java - the associated costs at each stage of the project are
                      > relatively larger to begin with, and grow more quickly as well.[/color]

                      I entirely agree with you, Dave. Moreover, I do have a mass of growing
                      but as-yet-unorganized notes, based mostly on experiences on large
                      projects I have consulted for or even been very intimately connected
                      with, showing why Python is superior to various plausible alternatives
                      (for various and different reasons in each case) for large-scale
                      software development, and what principles, practices and patterns best
                      enable teams in various conditions to actualize those advantages.

                      That is the book I want to write, the one I have always wanted to write;
                      the Nutshell and the Cookbook (and now their second editions) keep
                      delaying that plan, but, in a sense, that's good, because I keep
                      accumulating useful experiences to enrich those notes, and Python keeps
                      growing (particularly but not exclusively in terms of third-party
                      extensions and tools) in ways that refine and sometimes indeed redefine
                      some key aspects. To give a simple technical example: I used to have
                      substantial caveats in those notes cautioning readers to use multiple
                      inheritance in an extremely sparing, cautious way, due to traps and
                      pitfalls that made it fragile. Nowadays, with the advent of 2.3, most
                      of those traps and pitfalls have gone away (in the newstyle object
                      model), to the point that the whole issue can be reconsidered.

                      Anybody who has written serious technical books can gauge the amount of
                      work it takes to turn "a mass of yet-unorganized notes" into a real
                      book: it's _staggering_. I can't seriously undertake the task of making
                      my copious notes into a book until I can consider devoting at least half
                      of my time to it for a year -- this means no other books in the making,
                      _and_ a reduction in the amount of consulting, teaching, mentoring, etc,
                      that I do. The biggest general issue is that a book cannot be
                      _interactive_, _customized_ to the specific skills and interests of a
                      reader, in the way in which I can customize interactively the kind of
                      hands-on teaching, mentoring and consulting which I do for a specific
                      customer.

                      For a given customer, I can and do find out what kinds of areas they
                      believe their large projects will cover, what skills their people start
                      with (and what skills can they expect other people to start with in the
                      future, depending on expected turnover), the "political" and "social"
                      dynamics of the team -- is the kind of "customer involvement" that's the
                      crux of Extreme Programming wrt other kinds of Agile Development
                      feasible at all, at what cost, etc, for example -- and so on. I can
                      avoid spending substantial time and energy on issues which don't matter
                      to project A even though they may be crucial to most large projects --
                      believe it or not, SOME projects need no networking, others will never
                      directly interface to a relational database, etc, etc, even though these
                      days 9 large projects out of 10 will need to deal with both kinds of
                      issues; and GUI issues, especially for large projects which mostly deal
                      with web interfacing vs others which will need traditional GUIs, can be
                      even more divergent. And this amount of variety is just for the
                      _technical_ issues; the political/social/business-plan ones, people's
                      skills and backgrounds, etc, are even more diverse...

                      To make a book, I will have to find an organization that works for busy
                      readers who don't have the time or patience to read through long parts
                      connected to database issues if they're on one of the few projects that
                      don't care about databases, and so forth -- structure sections,
                      chapters, appendices, footnotes, sidebars, ... so that skimming or
                      skipping the "don't care about it right now" parts can work; find a way
                      to reach that part of the audience that has never really undertaken a
                      large project before, or has played in such projects the role of a "cog"
                      without a clear picture of the whole structure, _as well all_ the lead
                      architects and tech-savvy project managers.

                      Lakos did manage, and I admire him immensely for that. Robert Martin
                      has also done great work, though his books, while good, are (IMHO) never
                      _quite_ as excellent as his superb essays (don't get me wrong: I wish I
                      was half as good as Uncle Bob!-). Eric Raymond's "Art of Unix
                      Programming" is one of the most useful books for would-be architects of
                      software systems that I've ever laid my paws on -- I rate it as close to
                      the Mythical Man-Month, Design Patterns, Programming Pearls, and a few
                      of the many recent books on Extreme and other Agile methods (my personal
                      favorites of the crop are Scott Ambler's and Kent Beck's books).

                      However, none of these excellent books really addresses the questions
                      specific to the architecture, design, and development practices that
                      work best for dynamic VHLLs, and specifically for Python; so, I do
                      believe the book I dream to write is still needed (even though I might
                      be a grandfather by the time I'm done with it;-).

                      Meanwhile, to people and firms which aren't interested in retaining my
                      professional services, the best advice I can give -- after that of
                      studying the various books I have mentioned above (as well as good
                      Python books -- I like my own, but then, of course, I'm biased; I'd also
                      suggest others, such as Holden's, Pilgrim's, Hetland's, ...) -- is to
                      try something like:
                      <http://groups.google.com/groups?safe...thon*&as_uauth
                      ors=alex%20mart elli&lr=&hl=en>
                      as well as similar searches for the many other authors that contribute
                      so validly to the Python discussions, of course.

                      Somewhere or other, in my 8190 posts found by the above Google Groups
                      search, I have expressed (often more than once, and with different
                      nuances depending on the exact subject, apparent skills and interests of
                      other discussants, etc; as well as sometimes based on my changing ideas
                      on some sub-issue, or changes in Python and other tools and
                      technologies) a majority of the issues that I touch upon in that "mass
                      of notes". Of course, the stuff is yet more disorganized than said
                      notes; however, it _is_ written to be read and hopefully understood by
                      others, while most of said notes are written essentially "to myself", to
                      remind me of the huge variety of things that may need to be covered
                      regarding the huge variety of facets that make up the subject "Large
                      Scale System Architecture, Design, and Development Practices with
                      Python". Moreover, a majority of the 8190 posts are undoubtedly dealing
                      with subjects that aren't really related to LSSADDPP. Hey, there's
                      _got_ to be some advantage in retaining me, or reading my hopefully
                      future book, rather than combing through all my posts, no?-)

                      Seriously: one day do I hope to start putting up some parts of those
                      notes, mutated into intelligible text and organized into kind of
                      almost-essays, on my website -- fragments of said future book, but more
                      accessible and usable than the sheer morass of posts above-mentioned.
                      But don't hold your breath for _that_, any more than for the book; I've
                      been meaning to redo my site for _years_, and it just hasn't happened...
                      there's always something else that looks more interesting, either
                      intrinsically, and/or because of the little issue of money;-). Some
                      stuff (mostly presentations) you can find at www.strakt.com, which also
                      has important stuff written by Jacob Hallén and others.

                      People with lot of important and interesting things to teach, who have
                      managed to do a much better job than me at organizing their stuff on the
                      web, include for example Fredrik Lundh and Marc-Andre Lemburg. The
                      latter gave an hour-long talk this summer at Europython on the subject.
                      Unfortunately I can't easily find his presentation on
                      www.europython.org, nor Fredrik's, but I'm sure that an abler searcher
                      than me will manage, and they do have their own websites as well. In
                      any case, I'm sure that either of them could be (and often is, in their
                      respective professional practices) at least as effective as a teacher,
                      consultant or mentor, on large-scale software projects in Python, as me;
                      and the same applies no doubt to many others. In fact, the Python world
                      is blessed, in my opinion, with quite a number of excellent people who
                      might fill such roles -- one more reason to consider Python for
                      large-scale, mission-critical development, in fact!!!-)


                      Alex

                      Comment

                      • Alex Martelli

                        #12
                        Re: Is there a &quot;Large Scale Python Software Design&quot; ?

                        Stephen Waterbury <golux@comcast. net> wrote:
                        ...[color=blue]
                        > For something as complex as CAD/CAM, you will probably want to
                        > make maximum use of interfaces and adaptors, with minimal and
                        > very judicious application of classic inheritance hierarchies.
                        > I am *not* an expert on interfaces and adapters, but several
                        > of the gurus on this list are.[/color]

                        Heh -- funny enough, I did develop my ideas on protocol adaptation
                        mostly while working in the CAD area (as Senior Software Consultant to
                        what used to be Cad.Lab, and is now Think3, for over 10 years).

                        Our main implementation language, over time, moved from Fortran to C,
                        then to C++ -- but we did have our own proprietary scripting language,
                        and a growing amount of applications' functionality was coded in that
                        higher-level language. Interfaces (formalized or not) were of course a
                        given -- in the last few years I was there (and later when I worked as a
                        consultant for them), as the firm had moved to Windows as the only
                        platform for its products, mostly COM interfaces among components
                        (earlier, we had tried Corba, Java, and less formalized ones). The
                        Gof4's Design Patterns, and Lakos' Larce Scale C++ Software Design,
                        helped us crystallize our ideas and practices when they came out (I
                        devoured both avidly as soon as I could get my hands on them;-), but we
                        _had_ mostly gone that way already. But something was missing, and
                        Robert Martin's excellent essays (the Dependency Inversion Principle
                        first and foremost) helped BUT didn't quite solve that something...

                        Protocol Adaptation can, at least potentially. Try Eby's PyProtocols
                        for a taste (I may not agree with every one of Eby's design and
                        architectural choices, but nevertheless it seems to me that PyProtocols
                        is, today, the best implementation of Protocol Adaptation ideas).
                        Unfortunately, _that_ is when our choice of programming languages bit --
                        none of them, including our proprietary scripting language, had
                        introspection and dynamism enough to get anywhere near. Java perhaps
                        might, with much huffing and puffing, but we had put it aside after
                        extensive trials: too hard to interface our huge existing base of C++,
                        and rewriting stuff from C++ to Java would have been a nightmare without
                        templates (generic programming) in Java at the time -- even quite apart
                        from performance issues, the productivity gains with Java were not worth
                        the migration costs (for a single-platform software company, at least;
                        had we still been striving on multiple platforms, I guess it might have
                        been different:-).

                        Python (as Eby's work shows, for example) is fully adequate for Protocol
                        Adaptation (as, no doubt, would other modern VHLLs!)...


                        Alex

                        Comment

                        • Stephen Waterbury

                          #13
                          Re: Is there a &quot;Large Scale Python Software Design&quot; ?

                          Jonathan Ellis wrote:
                          [color=blue]
                          > ... A more-experienced co-worker pointed me in the right
                          > direction, and the IDE did the rest. ("Find definition," "Find
                          > references.") Grep can do much the same thing, but painfully slowly --
                          > and inaccurately, when you have a bunch of interfaces implementing the
                          > same method names. ...[/color]

                          Try "glimpse" (http://webglimpse.net) -- it uses a superset of
                          grep's arguments and can search large collections of files at
                          a single bound! Re-indexing takes a few seconds, but doesn't
                          need to be done unless there are major changes. The indexing
                          makes it considerably faster than grep (you can even read the
                          index into memory using glimpseserver, and then searches of
                          ~100MB of files take a fraction of a second). The first thing
                          I do when using any large Python library is put a glimpse
                          index on it.

                          Steve

                          Comment

                          • GerritM

                            #14
                            Re: Is there a &quot;Large Scale Python Software Design&quot; ?

                            "Jonathan Ellis" <jbellis@gmail. com> schreef in bericht
                            news:1098195361 .960550.257070@ z14g2000cwz.goo glegroups.com.. .
                            <..snip...>[color=blue]
                            > Almost four years ago I started working at a company with about 500
                            > kloc of Java code. Thanks largely to tool support I was able to get in
                            > and start fixing bugs my first day (this is without significant prior
                            > Java experience). A more-experienced co-worker pointed me in the right
                            > direction, and the IDE did the rest. ("Find definition," "Find
                            > references.") Grep can do much the same thing, but painfully slowly --
                            > and inaccurately, when you have a bunch of interfaces implementing the
                            > same method names. Even after years in the codebase, I still used
                            > these heavily; the codebase grew to about 800 kloc during the 3 years I
                            > worked there. Developers came and went; even if my memory were good
                            > enough to remember all the code _I_ ever wrote, I'd still have to
                            > periodically repeat the familiarization process with code written by
                            > others.
                            >[/color]
                            The point you make is that good tooling is important. I worked 12 years ago
                            in a large Objective-C environment. The same static versus dynamic wars were
                            raging at that time (Objective-C vs C++). I fully agree that good toold make
                            quite a difference. Most often very simple tools can do wonders. The dynamic
                            nature of Objective-C made also dynamic tools feasible, with an amazing
                            small extension. The run-time instrumentation proved at least as powerful,
                            as the compile time tools. Nowadays the same code is ported to Java, but
                            unfortunately the same powerful instrumentation is lost.
                            <...snip>[color=blue]
                            > -Jonathan[/color]

                            Contrary to your believe I would jump into larger scale Python development
                            without hesistation. However, I would introduce a few naming conventions to
                            support the static tool part.

                            kind regards, Gerrit
                            <www.extra.research.philips.com/ natlab/sysarch/>

                            --
                            Praktijk voor Psychosociale therapie Lia Charité
                            <www.liacharite .nl>



                            Comment

                            • Josiah Carlson

                              #15
                              Re: Is there a &quot;Large Scale Python Software Design&quot; ?

                              [color=blue]
                              > Heh. "Large" depends on a lot of things, particularly connectedness,
                              > but I really can't picture 10k being large under any circumstances.[/color]

                              Ok, so what is large? How many orders of magnitude larger than 10k
                              lines does it take for a piece of software to be large? And why should
                              you be the judge?

                              I'd let it slip to medium, but I wouldn't say that the project was small.
                              Small is something you can do in a weekend because you've been putting
                              it off. Small is something a newb to the language can do in a week
                              while they are learning the language.

                              - Josiah

                              Comment

                              Working...