Python advocacy in scientific computation

[Steve Holden]

Re: Python advocacy in scientific computation

Andy Salnikov wrote:[color=blue]
> "Michael Tobis" <mtobis@gmail.c om> wrote in message
> news:1141750637 .180979.69890@j 33g2000cwa.goog legroups.com...
>[color=green][color=darkred]
>>>Indentatio n
>>>makes all kinds of inlined code extremely clumsy or practically
>>>impossible
>>>in Python.[/color]
>>
>>This is the only sensible argument against the indentation thing I've
>>heard. Python squirms about being inlined in a presentation template.
>>Making a direct competitor to PHP in pure Python is problematic.
>>
>>While there are workarounds which are not hard to envision, it seems
>>like the right answer is not to inline small fragments of Python code
>>in HTML, which is probably the wrong approach for any serious work
>>anyway. This problem does, however, seem to interfere with adoption by
>>beginning web programmers, who may conceivably end up in PHP or perhaps
>>Perl Mason out of an ill-considered expedience.
>>
>>Why this should matter in this discussion, about scientific
>>programming , escapes me though.
>>
>>When you say "all kinds" of inlined code, do you have any other
>>examples besides HTML?
>>[/color]
>
> Makefiles is one example. Shell script containing snippet(s) of
> Python code is another one.
>
> At one time I also tried to make a simple "configurat ion file"
> engine based on Python for a big Framework used in one physics lab.
> Idea was to have a Python extension for that C++ framework and
> to configure the Framework from Python code, like:
>
> # Module means C++ Framework module, not Python
>
> Module1.param1 = "a string"
> Module2.paramX = [ 1, 2, 3 ]
> # etc., with all Python niceties.
>
> People who were using this Framework were all hard-core physicists,
> some of them knew Fortran, many were exposed to C++. There were
> few other "languages" , some of them home-grown, used for different
> tasks, but none of these mentioned languages ever placed so much
> significance on the whitespaces. There were some big surprises for
> people when they discovered they can't arbitrary indent pieces of
> the above configuration files because it is all Python code. Add
> here space/tabs controversy if it is not enough yet to confuse
> poor physicist fellows :) I think that config file project was killed
> later in favor of less restrictive format (I left the lab before that,
> can't say for sure.)
>[/color]
I just hope this remains a "someone made a poor choice of configuration
language and trained the users inadequately" story, and does not
transmogrify into a "Python is bad" story.

You mention makefiles and shell scripts as contexts unsympathetic to
Python's indentation requirements, but frankly you don't see much code
in any language except shell inlined in these contexts.

Given the makefile's requirement that significant leading whitespace be
tabs and not spaces and you have a recipe for disaster inlining any
language.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd www.holdenweb.com
Love me, love my blog holdenweb.blogs pot.com

[Michael Tobis]

Re: Python advocacy in scientific computation

I think I agree with Steve here.

I suspect you should either have sufficiently trained your users in
Python, or have limited them to one-line statements which you could
then strip of leading whitespace before passing them to Python, or even
offered the alternative of one or the other. This would not have been
much extra work.

As for shell scripts generating Python code, I am not sure what you
were trying to do, but if you're going that far why not just replace
the shell script with a python script altogether?

os.system() is your friend.

I also agree with Steve that I can't see what this has to do with
makefiles. (But then I think "make" is a thoroughly bad idea in the
first place, and think os.system() is my friend.)

mt

[Andy Salnikov]

Re: Python advocacy in scientific computation


"Steve Holden" <steve@holdenwe b.com> wrote in message
news:mailman.28 75.1141799254.2 7775.python-list@python.org ...[color=blue]
> Andy Salnikov wrote:[color=green]
>> "Michael Tobis" <mtobis@gmail.c om> wrote in message
>> news:1141750637 .180979.69890@j 33g2000cwa.goog legroups.com...
>>[color=darkred]
>>>
>>>When you say "all kinds" of inlined code, do you have any other
>>>examples besides HTML?
>>>[/color]
>>
>> Makefiles is one example. Shell script containing snippet(s) of
>> Python code is another one.
>>
>> At one time I also tried to make a simple "configurat ion file"
>> engine based on Python for a big Framework used in one physics lab.
>> Idea was to have a Python extension for that C++ framework and
>> to configure the Framework from Python code, like:
>>
>> # Module means C++ Framework module, not Python
>>
>> Module1.param1 = "a string"
>> Module2.paramX = [ 1, 2, 3 ]
>> # etc., with all Python niceties.
>>
>> People who were using this Framework were all hard-core physicists,
>> some of them knew Fortran, many were exposed to C++. There were
>> few other "languages" , some of them home-grown, used for different
>> tasks, but none of these mentioned languages ever placed so much
>> significance on the whitespaces. There were some big surprises for
>> people when they discovered they can't arbitrary indent pieces of
>> the above configuration files because it is all Python code. Add
>> here space/tabs controversy if it is not enough yet to confuse
>> poor physicist fellows :) I think that config file project was killed
>> later in favor of less restrictive format (I left the lab before that,
>> can't say for sure.)
>>[/color]
> I just hope this remains a "someone made a poor choice of configuration
> language and trained the users inadequately" story, and does not
> transmogrify into a "Python is bad" story.
>[/color]
It does not, and I did not say it's "bad". But people do percieve it
as at least very weird kind of language in a modern times of all the
"curly brace languages".
[color=blue]
> You mention makefiles and shell scripts as contexts unsympathetic to
> Python's indentation requirements, but frankly you don't see much code in
> any language except shell inlined in these contexts.
>[/color]
Shell's strength is in the process spawning/management and input/output
redirection, Python is rather weak in that area but OTOH Python is
strong in processing highly structured and numeric data, where shells
are really weak. I saw lots of awk or sed "code" embedded in scripts
so your claim that nothing except sheel is being inlined does not look
right to me.
[color=blue]
> Given the makefile's requirement that significant leading whitespace be
> tabs and not spaces and you have a recipe for disaster inlining any
> language.
>[/color]
I saw makefiles with thousands lines of Perl code in them. I agree this
(Perl) is disaster, but it would probably be better if it was Python code
instead.

Andy.

[Andy Salnikov]

Re: Python advocacy in scientific computation


"Michael Tobis" <mtobis@gmail.c om> wrote in message
news:1141837339 .598657.177260@ j52g2000cwj.goo glegroups.com.. .[color=blue]
>I think I agree with Steve here.
>
> I suspect you should either have sufficiently trained your users in
> Python, or have limited them to one-line statements which you could
> then strip of leading whitespace before passing them to Python, or even
> offered the alternative of one or the other. This would not have been
> much extra work.
>
> As for shell scripts generating Python code, I am not sure what you
> were trying to do, but if you're going that far why not just replace
> the shell script with a python script altogether?
>
> os.system() is your friend.
>
> I also agree with Steve that I can't see what this has to do with
> makefiles. (But then I think "make" is a thoroughly bad idea in the
> first place, and think os.system() is my friend.)
>
> mt
>[/color]

Actually os.system() is rather poor replacement for the shell's
capabilities, and it's _very_ low level, it's really a C-level code
wrapped in Python syntax. Anyway, to do something useful you need
to use all popen() stuff, and this is indeed infinitely complex
compared to the easy shell syntax.

Andy.

[Greg Ewing]

Re: Python advocacy in scientific computation

Andy Salnikov wrote:[color=blue]
> I saw lots of awk or sed "code" embedded in scripts[/color]

In my experience, embedding any of make/sh/awk/sed in
any of the others is a nightmare of singlequote/
doublequote/backslash juggling that makes a few
tab/space problems in Python pale by comparison.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand

http://www.cosc.canterbury.ac.nz/~greg

[Greg Ewing]

Re: Python advocacy in scientific computation

Andy Salnikov wrote:
[color=blue]
> Actually os.system() is rather poor replacement for the shell's
> capabilities, and it's _very_ low level, it's really a C-level code
> wrapped in Python syntax.[/color]

Since os.system() spawns a shell to execute the command,
it's theoretically capable of anything that the shell
can do. It's somewhat inelegant having to concatenate
all the arguments into a string, though.

I gather there's a new subprocess management module
coming that's designed to clean up the mess surrounding
all the popen() variants. Hopefully it will make this
sort of thing a lot easier.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand

http://www.cosc.canterbury.ac.nz/~greg

[Greg Ewing]

Re: Python advocacy in scientific computation

Peter Maas wrote:
[color=blue]
> This is hard to understand for an outsider. If you pass an int, a float,
> a string or any other "atomic" object to a function you have "pass by
> value" semantics. If you put a compound object like a list or a dictionary
> or any other object that acts as an editable data container you can return
> modified *contents* (list elements etc.) to the caller, exactly like in
> Java and different from C/C++.[/color]

There's really no difference here -- when you pass an
int, you're passing a pointer to an int object, just
the same as when you pass a list, you're passing a
pointer to a list object. It's just that Python
doesn't provide any operations for changing the
contents of an int object, so it's hard to see
the difference.

The similarity is brought out by the following
example:
[color=blue][color=green][color=darkred]
>>> def a(x):[/color][/color][/color]
.... x = 42
....[color=blue][color=green][color=darkred]
>>> def b(x):[/color][/color][/color]
.... x = [42]
....[color=blue][color=green][color=darkred]
>>> y = 3
>>> a(y)
>>> print y[/color][/color][/color]
3[color=blue][color=green][color=darkred]
>>> y = [3]
>>> b(y)
>>> print y[/color][/color][/color]
[3]

What this shows is that assignment to the parameter
*name* never affects anything outside the function,
regardless of whether the object passed in is mutable
or immutable.

It's best to avoid using terms like "by reference" when
talking about Python parameter passing, because it's
hard to tell whether the person you're talking to
understands the same thing by them. But if you
insist, the correct description in Algol terms is
that Python passes pointers to objects by value.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand

http://www.cosc.canterbury.ac.nz/~greg

[Michael McNeil Forbes]

Re: Python advocacy in scientific computation

Robert Kern <robert.kern@gm ail.com> writes:[color=blue]
> sturlamolden wrote:[/color]
....[color=blue][color=green]
> > 5. Versioning control? For each program there is only one
> > developer and a single or a handful users.[/color][/color]
....[color=blue]
> This is one thing that a lot of people seem to get wrong: version
> control is not a burden on software development. It is a great
> enabler of software development. It helps you get your work done
> faster and easier even if you are a development team of one. You can
> delete code (intentionally! ) because it's not longer used in your
> code, but you won't lose it. You can always look at your history and
> get it again. You can make sweeping changes to your code, and if
> that experiment fails, you can go back to what was working
> before. Now you can do this by making copies of your code, but
> that's annoying, clumsy, and more effort than it's worth. Version
> control makes the process easier and lets you do more interesting
> things.
>
> I would go so far as to say that version control enables the
> application of the scientific method to software development. When
> you are in lab, do you say to yourself, "Nah, I won't write anything
> in my lab notebook. If the experiment works at the end of the day,
> only that result matters"?[/color]
....
A slightly off topic note:

I find that version control (VC) has many advantages for
scientific research (I am a physicist).

1) For software as Robert mentions I find it indispensable.
2) Keeping track of changes to papers (as long as they are plain text
like LaTeX). This is especially useful for collaborations: using
the diff tools one can immediately see any changes a coauthor may
have made.

(I even use branching: maintaining one branch for the journal
submission which typically has space restrictions, and another for
preprint archives which may contain more information.)
3) Using VC allows you to easily bring another computer up to date
with your current work. If I go to a long workshop and use local
computing resources, I simply checkout my current projects and I
can work locally. When I am done, I check everything back in and
when I get home, I can sync my local files.
-------

Another aspect of python I really appreciate are the unit testing
facilities. The doctest, unittest, and test modules make it easy to
include thorough tests: crucial for making sure that you can trust the
results of your programs. Organizing these tests in MATLAB and with
other languages was such a pain that I would often be tempted to omit
the unit tests and just run a few simulations, finding errors on the
fly.

Now I almost always write unit tests along with---or sometimes
before---I write the code.

Michael.

[Philip Austin]

Re: Python advocacy in scientific computation

Michael McNeil Forbes <mforbes@lnsDOT mit.edu> writes:[color=blue]
>
> I find that version control (VC) has many advantages for
> scientific research (I am a physicist).
>[/color]

Greg Wilson also makes that point in this note:

http://www.nature.com/naturejobs/2005/050728/full/nj7050-600b.html

Where he describes his excellent (Python Software Foundation sponsored)
course on software carpentry for scientists:

http://www.third-bit.com/swc2/index.html

Regards, Phil