A story about Python... sort of

[Paul Boddie]

Re: A story about Python... sort of

"Max Khesin" <max@cNOvSisiPo nAtecMh.com> wrote in message news:<dGZMa.517 2$351.1725266@t wister.nyc.rr.c om>...[color=blue]
>
> I am not sure where we disagree. This is exactly my point. The statement
> "
> Engineering Lessons
> -------------------
> 1. C/C++ is no longer a viable development language
> "
> is pure rubbish. C++ is still great for certain kinds of projects, and there
> are lots of open-source and proprietary projects to prove this.[/color]

I wouldn't agree with you unreservedly here. In many respects, the
choice of C++ for projects is often an educational problem with the
developers - people choose it because it's what they know, potentially
not very well in many cases. So one could say that it's really
something they just know something about - it seems like the
right/safe choice, presumably because their peers/acquaintances who
are just as badly informed tell them so.

Those of us who are used to more high-level languages would think
twice about writing a large application using a language/library
combination without decent (ie. modern) support for memory management,
for example. I can imagine that many developers find it challenging
and even rewarding to think up interesting schemes for allocating and
freeing memory - perhaps they even think that this (and lots of other
unnecessary wheel reinvention) is what programming is all about.
Personally, I'd rather get on with implementing the actual system
concerned.

So, I'd rephrase the original statement: C/C++ are frequently
suboptimal choices for application development. Why? Poor support for
near-essential features found in contemporary languages combined with
the absence of timely, effective standardisation of useful library
functionality.

Paul

[John J. Lee]

Re: A story about Python... sort of

aahz@pythoncraf t.com (Aahz) writes:
[color=blue]
> In article <Yg3Na.97773$R7 3.11565@sccrnsc 04>,
> Russell Reagan <rreagan@attbi. com> wrote:[/color]
[...][color=blue]
> At the same time, more and more of those games are switching to using
> C/C++ only for the rendering engine and using a scripting language (Lua
> or Python) for the gameplay itself.[/color]

Is this true of big-$ commercial games? What sort of market share do
high-level / interpreted languages have there?

Never having been a (graphical) games player, I don't know the first
thing about how games developers do things.


John

[Aahz]

Re: A story about Python... sort of

In article <87llve8fh3.fsf @pobox.com>, John J. Lee <jjl@pobox.co m> wrote:[color=blue]
>aahz@pythoncra ft.com (Aahz) writes:[color=green]
>>
>> At the same time, more and more of those games are switching to using
>> C/C++ only for the rendering engine and using a scripting language (Lua
>> or Python) for the gameplay itself.[/color]
>
>Is this true of big-$ commercial games? What sort of market share do
>high-level / interpreted languages have there?[/color]

Depends what you mean by big-$. Humongous Entertainment has recently
switched to requiring Python for all new games. Lua is even more
prevalent; see http://www.lua.org/uses.html
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

Usenet is not a democracy. It is a weird cross between an anarchy and a
dictatorship.

[Peter Hansen]

Re: A story about Python... sort of

Aahz wrote:[color=blue]
>
> In article <87llve8fh3.fsf @pobox.com>, John J. Lee <jjl@pobox.co m> wrote:[color=green]
> >aahz@pythoncra ft.com (Aahz) writes:[color=darkred]
> >>
> >> At the same time, more and more of those games are switching to using
> >> C/C++ only for the rendering engine and using a scripting language (Lua
> >> or Python) for the gameplay itself.[/color]
> >
> >Is this true of big-$ commercial games? What sort of market share do
> >high-level / interpreted languages have there?[/color]
>
> Depends what you mean by big-$. Humongous Entertainment has recently
> switched to requiring Python for all new games. Lua is even more
> prevalent; see http://www.lua.org/uses.html[/color]

Well, you can't get any bigger than "humongous" , can you? ;-)

[Dave Brueck]

Re: A story about Python... sort of

On Thursday 03 July 2003 11:48 pm, Russell Reagan wrote:[color=blue]
> "Dave Brueck" <dave@pythonapo crypha.com> wrote
>[color=green][color=darkred]
> > > I
> > > mean, is Linux (or Windows) 'not a viable project'??[/color]
> >
> > Well, again, neither of those are "applicatio ns level" projects.[/color]
>
> There is a rather large industry which I'll call "computer games" that is
> rather CPU intensive, and AFAIK the vast majority of those games are
> written in C/C++. It really depends on the definition of "applicatio n
> level". If we're only including things like word processors and web
> browsers in the "applicatio n level", then there isn't a great need for C++
> in the
> "applicatio n level",[/color]

Actually, games are a particularly good example to illustrate the point:

1) In the movement away from a lower-level language, games are probably one of
the last hold-outs since performance often means so much. Still, even games
do make the transition - the transition away from assembly being the main
example.

2) Even the most performance-intensive games of today are already
transitioning towards higher-level languages - is there any major
first-person shooter or real-time strategy game coming out these days that
doesn't boast a powerful scripting language? With each new generation of
games the developers try to push more and more of the functionality into
their scripting engine leaving as little as possible behind in C/C++ - the
render loop, *some* of the AI, etc. Not only is the game customizable by the
customers, the developers themselves prefer it because of fewer bugs and it
makes it much easier to try new and cool stuff out.

3) More and more of the performance-intensive work is handled by hardware
nowadays anyway - both video and audio. Furthermore, the game itself usually
relies on a pretty rich and powerful supporting library like OpenGL or even
DirectX, which depending on the route you take can supply a ton of the
functionality that the game developer would normally write in C or C++.

4) All of the above mean that in many cases you already *can* do some pretty
elaborate games in higher-level languages (the games listed on Pygame are a
great example), and there's every indication that the trend will continue.

There came a time when it was no longer economically viable to develop an
entire game in assembly, and IMO we've *already* passed the point where it's
no longer economically viable to develop most games entirely in C++ - the
time to market is too long, it's too expensive to add new features,
recovering from early incorrect decisions is too costly, etc. Games are
moving to higher-level languages as much as they can because it's a
competitive advantage to do so.
[color=blue]
> but there are certainly areas where speed and memory
> efficiency is important.[/color]

Oh, nobody disagrees with that. But due to increases in efficiency and
decreases in prices, the number of cases is shrinking wherein speed and
memory constraints require you to drop to a lower-level language.

-Dave

[Ben Finney]

Re: A story about Python... sort of

On Mon, 07 Jul 2003 02:02:06 GMT, Russell Reagan wrote:[color=blue]
> Anyway, I know of one chess program written in python, and it is
> dreadfully slow compared to *any* program written in C/C++ (that I've
> seen anyway).[/color]

Have you looked at the code for it? Have you profiled it to see where
its bottlenecks are? It's often the case that a program is low because
of poor design, or simply choosing a slow algorithm.

Until profiling, of course, you can't know which algorithms are too slow
for the program.
[color=blue]
> The things that make a chess program fast aren't really python's
> strengths, but hopefully I'm being pessimistic due to my lack of
> python/c knowledge.[/color]

My suggestion for those who want to write programs that do something
quickly:

- Write a program that does the job at all, paying attention to
simplicity and readability and *no* attention to optimisation.

- Debug the program so it does everything it's supposed to do, albeit
slowly.

- Once the program is correct, and not before, profile it to see where
it's slow.

- Pick the biggest bottleneck and make it faster, by one or more of:

- Choose a faster algorithm, perhaps sacrificing readability or
simplicity
- Re-implement in C

- Iterate the previous step until the program is fast enough or you
run out of (time|money).

You'll have a program that is quite readable, except in the places where
it needs to be fast. In my experience, those places are surprisingly
few, and are surprisingly different to where you expected them to be.
[color=blue]
> import chess
> chess.run_progr am_in_c
> print "Thanks for playing! Bye!"[/color]

It may well be that all the "real" chess-thinking stuff may be too slow
in Python; you might end up with the move-evaluation routine in C, for
example.

But discover that by profiling a Python implementation first! If it
turns out to be too slow, at least you've debugged a working algorithm,
and can treat it as pseudocode for the port to C. If it turns out that
the bottlenecks are somewhere else entirely, you've saved yourself a
huge misguided optimisation effort.

--
\ "My roommate got a pet elephant. Then it got lost. It's in the |
`\ apartment somewhere." -- Steven Wright |
_o__) |
http://bignose.squidly.org/ 9CFE12B0 791A4267 887F520C B7AC2E51 BD41714B

[Russell Reagan]

Re: A story about Python... sort of

I didn't investigate the python chess program a great deal, since I'm still
learning the language, but it is very likely that it wasn't an efficient
implementation. It takes years to learn how a chess program works, and to
learn all of the tricks you can do to speed things up (and even then you
have to have the programming skill to implement those things).

I agree that it is not good to worry about optimization prematurely, but
computer chess might be a bit of a unique area. Computer chess is the most
heavily researched area of artificial intelligence ever, and as such, there
are tons of "standard" low level tricks or data representations that you can
use to your advantage. For instance, you can use board representation A and
get thing B for "free" (B might be fast attack detection, or the ability to
evaluate some complex aspect of a position quickly). When designing a chess
engine, you have to decide what you want to do, then use the data
representation that gives you the most things you want to do for "free".

This is also why OOP might not be of much use for chess programming. The
goal of OOP is to write correct code by being able to more easily make
modifications to the code by data hiding, writing generalized routines, etc.
Computer chess is so heavily researched, that there are several "standard"
board representations that people use, and they're low level, involving lots
of bit fiddling. Making a board object that doesn't know how a piece object
is implemented serves little purpose other than to slow things down. Part of
a fast chess program is that every part knows how everything else is
implemented, and exploiting that. I'm not discounting anything you said. I'm
just saying that computer chess may be one of the few areas where the
generally good advice doesn't always apply, due to the ridiculous amount of
research. Heck, computer chess was being researched before computers even
existed!
[color=blue]
> - Write a program that does the job at all, paying attention to
> simplicity and readability and *no* attention to optimisation.[/color]

In general, this is good advice, but for reasons explained above, paying no
attention to optimization might be the equivalent of choosing a bad design
for this specific problem. By not paying any attention to efficiency early
on, you effectively limit your top speed later on.
[color=blue]
> - Once the program is correct, and not before, profile it to see where
> it's slow.[/color]

Since computer chess is so heavily researched, I can tell you that the area
consuming the biggest chunk of time will be position evaluation, unless your
program doesn't use a lot of the well known tricks for things like attack
detection, in which case it might spend a ton of time doing that, ensuring
that a move doesn't leave your king in check, that kind of stuff. There is
really not a lot you can do to improve either of those short of starting
over from scratch and creating your data structures in such a way as to
exploit lots of low level tricks.
[color=blue]
> at least you've debugged a working algorithm,
> and can treat it as pseudocode for the port to C.[/color]

For reasons explained above, I'm not sure that you can make the direct
translation between "python pseudocode" to "C/C++". It sounds like you're
thinking, "I'll just implement this python program in C/C++, and then it
will be faster," but I'm not sure that is the case. Sure it will be faster,
but probably still pretty slow. Making a really fast chess program would
require changing the data structures significantly, and I'm not sure what
use the python code would be after you made a big overhaul of the data
representations .

What do you think? I may be way off. Maybe I'm putting too much emphasis on
speed.

[Dave Brueck]

Re: A story about Python... sort of

On Monday 07 July 2003 02:02 am, Russell Reagan wrote:[color=blue][color=green]
> > There came a time when it was no longer economically viable to develop an
> > entire game in assembly, and IMO we've *already* passed the point where
> > no longer economically viable to develop most games entirely in C++ - the
> > time to market is too long, it's too expensive to add new features,
> > recovering from early incorrect decisions is too costly, etc. Games are
> > moving to higher-level languages as much as they can because it's a
> > competitive advantage to do so.[/color]
>
> After thinking about it I tend to agree. I'm a little partial to C/C++
> because I develop (as a hobby) a chess playing program, and I like
> tinkering with it to see how fast it can go, and finding clever ways of
> doing things. A chess program is different from a 3D game in that with a 3D
> game, you can stop at some point and say, "ok, this is fast enough." There
> is little point in making the game run at 1000 frames per second if no
> human eye can see more than 60 (or whatever the number is).[/color]

True, although rather than let the frame rate continue up the game designers
will simply add more polygons, more actors, more textures, greater viewable
distance, etc. so they won't ever reach the "fast enough" point either.
Instead you always reach the "fast enough for now" limit.

There will remain *some* cases where you need a lower-level language in order
to get more speed, but those cases are becoming less frequent - for each one
there comes a time when the cost of that extra speed becomes too much. Also,
some projects (like Pysco) open the possibility of on-par (or even greater,
in theory) speed by using a higher-level language.

But my main point was that, at least for the stuff I do, there's never any
point in writing all or even most of an application in C++ anymore (and the
few parts that do need more speed would be in a dumbed-down, simplified C++
at that). If I were writing the chess game I'd probably write the entire
thing in Python to begin with anyway, just to get it working. Having a slow,
but working app would either help maintain my interest in the project or let
me know early on that it's not something I really want to do.

If I kept at it then I could, at some point, move some speed-critical code to
C or C++, but I'd try to avoid it for as long as possible. For example, you
mentioned that your interest was at least in in part about finding ways to do
things faster or to make the game smarter. In a higher-level language like
Python that sort of expermimentatio n is relatively cheap - in C++ you might
avoid radical experiments because the cost of change is too high. Your
biggest speed and smartness improvements after picking the low-hanging fruit
are likely to be algorithmic in nature (e.g. using a more ingenious heuristic
to reduce the search space of next moves) as opposed to low-level tuning of
loops or data storage, so it would be advantageous to do that work at as high
a level as possible (and you can always translate those algorithmic
optimizations to a lower-level language someday if you want).

Just my two cents of course,
-Dave