Running autogenerated code in another python instance

**Bengt Richter** · Nov 2 '05, 07:45 AM

Re: Running autogenerated code in another python instance

On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <cochrane@shake 56.esscc.uq.edu .au> wrote:
[color=blue]
>Hi all,
>
>I've got an application that I'm writing that autogenerates python code
>which I then execute with exec(). I know that this is not the best way to
>run things, and I'm not 100% sure as to what I really should do. I've had a
>look through Programming Python and the Python Cookbook, which have given me
>ideas, but nothing has gelled yet, so I thought I'd put the question to the
>community. But first, let me be a little more detailed in what I want to
>do:
>[/color]
[...][color=blue]
>
>Any help or advice would be really (really!) appreciated.
>[/color]
It's a little hard to tell without knowing more about your
user input (command language?) syntax that is translated to
or feeds the process that "autogenera tes python code".

E.g., is it a limited python subset that you are accepting as input,
or a command language that you might implement using the cmd module, or ?
There are lots of easy things you could do without generating and exec-ing
python code per se. How complex is a user session state? What modifies it?
What actions are possible? History? Undo? Is the data a static passive
resource to view, or partly generated or made accessible by prior actions
in a session? Single user or collaborative? Shared access to everything or
only to static data? Etc., etc.

Some examples of user input and corresponding generated python might help,
with some idea of the kind of visualization aspects being controlled ;-)

Regards,
Bengt Richter

**Do Re Mi chel La Si Do** · Nov 2 '05, 07:45 AM

Re: Running autogenerated code in another python instance

Hi!

I did not understand anything with your chatterer message.
But, perhaps, the code below will help you.

@-salutations

Michel Claveau

def runcode(srcode) :
import sys,traceback
sret=True
try:
ccod=compile(sr code, 'Paul-code', 'exec')
flagcompile=Tru e
except:
print "compilo-error"
flagcompile=Fal se
if flagcompile==Tr ue:
try:
exec(ccod,globa ls(),globals())
sret=True
except:
tb=sys.exc_info ()[2] #for traceback
print str(sys.exc_inf o()[0])
print traceback.forma t_exc()
sret=False
return sret

s="""global c
a=1000
b=123
c=a+b"""

if runcode(s):
print c

**Do Re Mi chel La Si Do** · Nov 2 '05, 07:55 AM

Re: Running autogenerated code in another python instance

With good spaces :

def runcode(srcode) :
import sys,traceback
sret=True
try:
ccod=compile(sr code, 'Paul-code', 'exec')
flagcompile=Tru e
except:
print "compilo-error"
flagcompile=Fal se
if flagcompile==Tr ue:
try:
exec(ccod,globa ls(),globals())
sret=True
except:
tb=sys.exc_info ()[2] #for traceback
print str(sys.exc_inf o()[0])
print traceback.forma t_exc()
sret=False
return sret

s="""global c
a=1000
b=123
c=a+b"""
if runcode(s):
print c

s="""global c
a=1000
b=123
c=aaa+b"""
if runcode(s):
print c

**Paul Cochrane** · Nov 3 '05, 04:35 AM

Re: Running autogenerated code in another python instance

On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
[color=blue]
> On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <cochrane@shake 56.esscc.uq.edu .au> wrote:
>[color=green]
>>Hi all,
>>
>>I've got an application that I'm writing that autogenerates python code
>>which I then execute with exec(). I know that this is not the best way to
>>run things, and I'm not 100% sure as to what I really should do. I've had a
>>look through Programming Python and the Python Cookbook, which have given me
>>ideas, but nothing has gelled yet, so I thought I'd put the question to the
>>community. But first, let me be a little more detailed in what I want to
>>do:
>>[/color][/color]

Bengt,

Thanks for your reply!
[color=blue]
> It's a little hard to tell without knowing more about your
> user input (command language?) syntax that is translated to
> or feeds the process that "autogenera tes python code".[/color]
Ok, I'll try and clarify things as much as I can.
[color=blue]
> E.g., is it a limited python subset that you are accepting as input,
> or a command language that you might implement using the cmd module, or ?[/color]
It's basically just a command language I guess. Perhaps it's best to
explain using an example. Here's the pyvisi code that generates a very
simple line plot using the vtk renderer module:

"""
Example of plotting lines with pyvisi
"""

# set up some data to plot
from Numeric import *

x = arange(10, typecode=Float)
y = x**2

# example code for how a user would write a script in pyvisi
from pyvisi import * # base level visualisation stuff
# import the objects to render the scene using the specific renderer
#from pyvisi.renderer s.gnuplot import * # gnuplot
from pyvisi.renderer s.vtk import * # vtk
#from pyvisi.renderer s.plplot import * # plplot

# define the scene object
# a Scene is a container for all of the kinds of things you want to put
# into your plot for instance, images, meshes, arrow/vector/quiver plots,
# contour plots, spheres etc.
scene = Scene()

# create a LinePlot object
plot = LinePlot(scene)

# add some helpful info to the plot
plot.title = 'Example 2D line plot'
plot.xlabel = 'x'
plot.ylabel = 'x^2'

plot.linestyle = 'lines'

# assign some data to the plot
plot.setData(x, y)

# render the scene to screen
scene.render(pa use=True, interactive=Tru e)

# save the scene out to file
## png
plot.setData(x, y) # have to do this now because we've already
# render()ed the scene, will be removed in the
# future
scene.save(fnam e="simpleLinePl ot.png", format=PngImage ())

This code then gets translated by the pyvisi module into the vtk-python
code:

import vtk
from Numeric import *
# LinePlot.__init __()
_plot = vtk.vtkXYPlotAc tor()
_renderer = vtk.vtkRenderer ()
_renderWindow = vtk.vtkRenderWi ndow()
_renderWindow.A ddRenderer(_ren derer)
_renderWindow.S etSize(640,480)
_renderer.SetBa ckground(1,1,1)
# Renderer._initR endererModule
# LinePlot.setDat a()
_x = array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0])
_xData = vtk.vtkDataArra y.CreateDataArr ay(vtk.VTK_FLOA T)
_xData.SetNumbe rOfTuples(len(_ x))
_y0 = array([0.0, 1.0, 4.0, 9.0, 16.0, 25.0, 36.0, 49.0, 64.0, 81.0])
_y0Data = vtk.vtkDataArra y.CreateDataArr ay(vtk.VTK_FLOA T)
_y0Data.SetNumb erOfTuples(len( _y0))
for i in range(len(_x)):
_xData.SetTuple 1(i,_x[i])
for i in range(len(_x)):
_y0Data.SetTupl e1(i,_y0[i])
_fieldData0 = vtk.vtkFieldDat a()
_fieldData0.All ocateArrays(2)
_fieldData0.Add Array(_xData)
_fieldData0.Add Array(_y0Data)
_dataObject0 = vtk.vtkDataObje ct()
_dataObject0.Se tFieldData(_fie ldData0)
_plot.AddDataOb jectInput(_data Object0)
_plot.SetXValue sToValue()
_plot.SetDataOb jectXComponent( 0,0)
_plot.SetDataOb jectYComponent( 0,1)
_plot.GetXAxisA ctor2D().GetPro perty().SetColo r(0, 0, 0)
_plot.GetYAxisA ctor2D().GetPro perty().SetColo r(0, 0, 0)
_renderer.SetBa ckground(1.0, 1.0, 1.0)
_lut = vtk.vtkLookupTa ble()
_lut.Build()
_colours = []
_colours.append (_lut.GetColor( 0))

_plot.SetPlotCo lor(0, _colours[0][0], _colours[0][1], _colours[0][2])
_plot.SetPositi on(0.1, 0.1)
_plot.SetWidth( 0.8)
_plot.SetHeight (0.8)
# Scene.render()
_lut = vtk.vtkLookupTa ble()
_refLut = vtk.vtkLookupTa ble()
_lut.Build()
_refLut.Build()
for _i in range(256):
_lut.SetTableVa lue(_i, _refLut.GetTabl eValue(255-_i))
_iRenderer = vtk.vtkRenderWi ndowInteractor( )
_iRenderer.SetR enderWindow(_re nderWindow)
# LinePlot.render ()
_renderer.AddAc tor2D(_plot)
_plot.SetTitle( 'Example 2D line plot')
_plot.SetXTitle ('x')
_plot.SetYTitle ('x^2')
_renderWindow.R ender()
_iRenderer.Star t()

Which is pretty ugly for the user to have to get to grips with, so I'm
trying to simplify the interface to the back end. I'm writing other
backends so that one only needs to change one line of code to then use
gnuplot or plplot as the renderer to generate the plot. Of course some
renderers can do things others can't, but that's another issue...

[color=blue]
> There are lots of easy things you could do without generating and exec-ing
> python code per se.[/color]
I'd love to know of other options. I like the idea of generating the code
one would have to write for a particular renderer so that if the user wanted
to, they could use the autogenerated code to form the basis of a specific
visualisation script they could then hack themselves. I also like it
because I can see how the pyvisi code maps directly to the back end.
[color=blue]
> How complex is a user session state?[/color]
Not 100% sure. My intention is that this is just a simple matter of
translating the commands specified at the high level into the lower level
stuff in the background. It can handle plotting several different graphs in
a loop as the data used to generate the graph changes, and it can handle
interactive display of the graphics in vtk (a vtk feature really, via
OpenGL), but nothing too tricky there. exec() is handy for running
commands generated on the fly, however, it doesn't keep the state of the
interpreter and so any variables generated on a previous call are lost.
Also, exec ignores globals etc (at least AFAIK). What would be great
would be to run the generated commands within the same scope and state,
then I wouldn't need to worry about this separate thread/process idea. I
just don't know how to do that.
[color=blue]
> What modifies it?
> What actions are possible? History? Undo?[/color]
There's no history or undo, just processing of the script.
[color=blue]
> Is the data a static passive
> resource to view, or partly generated or made accessible by prior actions
> in a session?[/color]
The data can be generated by prior actions in a given python script, but
basically pyvisi is set up to grab the data it's passed, and then work out
how best to render it. So in some sense the data is static and passive,
however can change at some later date, but one has to recall the setData()
method and then the render() method of the scene object to process any
changes that have occured in the data.
[color=blue]
> Single user or collaborative?[/color]
Definitely single user.
[color=blue]
> Shared access to everything or
> only to static data? Etc., etc.[/color]
Preferably only shared access to static data, but I do some variable name
mangling to try and keep autogenerated variables separate to anything the
user has created themselves.
[color=blue]
> Some examples of user input and corresponding generated python might help,
> with some idea of the kind of visualization aspects being controlled ;-)[/color]
With my initial post I didn't want to put in examples as it was already
quite long and I didn't want to put people off replying. btw: I really
appreciate your feedback; I usually work best bouncing ideas off people to
find a clear way to do something.

One of the main ideas of the module is to distill the common visualisation
tasks down to a simple set of commands, and then let the interface work out
how to actually implement that. One wants to be able to put titles, and
labels and axes on things, one wants to plot 3D surfaces of data, arrows of
vector fields, ellipses of tensor data, isosurfaces of 3D scalar data, and
be able to display this to the screen and save to file. Other visualisation
aspects to be controlled are such things as camera angles, number of
contours in a plot, different kinds of colourmaps, etc.

Just to try and be of some help, here's another example of plotting some
data and its output. This time it's plotting of a random vector field with
arrows in three dimensions. First the pyvisi code:

"""
Example of plotting a 3D vector field with pyvisi
"""

# set up some data to plot
from Numeric import *

dim = 10

# initialise the positions of the vectors
x = zeros((dim,dim) , typecode=Float)
y = zeros((dim,dim) , typecode=Float)
z = zeros((dim,dim) , typecode=Float)

# initialise the vector displacements
# (I may need to rethink how this works in the interface)
dx = zeros((dim,dim) , typecode=Float)
dy = zeros((dim,dim) , typecode=Float)
dz = zeros((dim,dim) , typecode=Float)

# set the positions randomly, and set the displacements to some smaller
# random number but of mean zero instead of distributed between 0 and 1
import random
random.seed()
for i in range(dim):
for j in range(dim):
x[i,j] = random.random()
y[i,j] = random.random()
z[i,j] = random.random()
dx[i,j] = (random.random( )-0.5)/5.0
dy[i,j] = (random.random( )-0.5)/5.0
dz[i,j] = (random.random( )-0.5)/5.0

# example code for how a user would write a script in pyvisi
from pyvisi import * # base level visualisation stuff
# import the objects to render the scene using the specific renderer
#from pyvisi.renderer s.gnuplot import * # gnuplot
from pyvisi.renderer s.vtk import * # vtk
#from pyvisi.renderer s.povray import * # povray

# define the scene object
# a Scene is a container for all of the kinds of things you want to put
# into your plot for instance, images, meshes, arrow/vector/quiver
# plots,
# contour plots, spheres etc.
scene = Scene()

# create a ArrowPlot3D object
plot = ArrowPlot3D(sce ne)

# add some helpful info to the plot
plot.title = 'Example 3D arrow/quiver/vector field plot'
plot.xlabel = 'x'
plot.ylabel = 'y'
plot.zlabel = 'z'

# assign some data to the plot
plot.setData(x, y, z, dx, dy, dz)

# render the scene to screen
scene.render(pa use=True, interactive=Tru e)

# save the scene out to file
plot.setData(x, y, z, dx, dy, dz) # have to do this because we've already
# render()ed the scene. This requirement
# will be removed in the future
scene.save(fnam e="arrowPlot3D. png", format=PngImage ())

And now the generated vtk-python code:

import vtk
from Numeric import *
_renderer = vtk.vtkRenderer ()
_renderWindow = vtk.vtkRenderWi ndow()
_renderWindow.A ddRenderer(_ren derer)
_renderWindow.S etSize(640,480)
_renderer.SetBa ckground(1,1,1)
# Renderer._initR endererModule
# ArrowPlot3D.__i nit__()
# ArrowPlot3D.set Data()
_x = array([0.877228328994, 0.412795474983, 0.163560730946, 0.89780023349,
0.129244456665, 0.40180234598, 0.542135110974, 0.46894547296,
<snip>
_y = array([0.750790237727, 0.355648864325, 0.659517997105, 0.0381646378866 ,
<snip>
_z = array([0.26125343592, 0.941964065844, 0.510850248162, 0.131409524918,
<snip>
_dx = array([0.0474228215121 , -0.0940162717243 , -0.0926294230912 ,
<snip>
_dy = array([0.0853229942053 , 0.0358464500795 , -0.0046119895345 7,
<snip>
_dz = array([-0.0559212949124 , 0.0827739088553 , 0.0849306644324 ,
<snip>
_points = vtk.vtkPoints()
_points.SetNumb erOfPoints(100)
_points.InsertP oint(0, 0.877228, 0.750790, 0.261253)
<snip>
_vectors = vtk.vtkFloatArr ay()
_vectors.SetNum berOfComponents (3)
_vectors.SetNum berOfTuples(100 )
_vectors.SetNam e("vectors")
_vectors.Insert Tuple3(0, 0.047423, 0.085323, -0.055921)
<snip>
_grid = vtk.vtkUnstruct uredGrid()
_grid.SetPoints (_points)
_grid.GetPointD ata().AddArray( _vectors)
_grid.GetPointD ata().SetActive Vectors("vector s")
# Scene.render()
_lut = vtk.vtkLookupTa ble()
_refLut = vtk.vtkLookupTa ble()
_lut.Build()
_refLut.Build()
for _i in range(256):
_lut.SetTableVa lue(_i, _refLut.GetTabl eValue(255-_i))
_iRenderer = vtk.vtkRenderWi ndowInteractor( )
_iRenderer.SetR enderWindow(_re nderWindow)
# ArrowPlot3D.ren der()
_arrow = vtk.vtkArrowSou rce()
_maxNorm = _grid.GetPointD ata().GetVector s().GetMaxNorm( )
_glyph = vtk.vtkGlyph3D( )
_glyph.ScalingO n()
_glyph.SetScale ModeToScaleByVe ctor()
_glyph.SetColor ModeToColorByVe ctor()
_glyph.SetScale Factor(0.1/_maxNorm)
_glyph.SetInput (_grid)
_glyph.SetSourc e(_arrow.GetOut put())
_glyph.Clamping Off()
_stripper = vtk.vtkStripper ()
_stripper.SetIn put(_glyph.GetO utput())
_lut = vtk.vtkLookupTa ble()
_lut.Build()
_refLut = vtk.vtkLookupTa ble()
_refLut.Build()
for i in range(256):
_lut.SetTableVa lue(i, _refLut.GetTabl eValue(255-i))
_mapper = vtk.vtkPolyData Mapper()
_mapper.SetInpu t(_stripper.Get Output())
_mapper.SetScal arRange(0, _maxNorm)
_actor = vtk.vtkActor()
_actor.SetMappe r(_mapper)
_renderer.AddAc tor(_actor)
_font_size = 14
_textProp = vtk.vtkTextProp erty()
_textProp.SetFo ntSize(_font_si ze)
_textProp.SetFo ntFamilyToArial ()
_textProp.BoldO ff()
_textProp.Itali cOff()
_textProp.Shado wOff()
_textProp.SetCo lor(0,0,0)
_titleMapper = vtk.vtkTextMapp er()
_titleMapper.Se tInput("Example 3D arrow/quiver/vector field plot")
_titleProp = _titleMapper.Ge tTextProperty()
_titleProp.Shal lowCopy(_textPr op)
_titleProp.SetJ ustificationToC entered()
_titleProp.SetV erticalJustific ationToTop()
_titleProp.SetF ontSize(18)
_titleActor = vtk.vtkTextActo r()
_titleActor.Set Mapper(_titleMa pper)
_titleActor.Get PositionCoordin ate().SetCoordi nateSystemToNor malizedDisplay( )
_titleActor.Get PositionCoordin ate().SetValue( 0.5, 0.95)
_renderer.AddAc tor(_titleActor )
_axes = vtk.vtkCubeAxes Actor2D()
_axes.SetCamera (_renderer.GetA ctiveCamera())
_axes.SetFlyMod eToOuterEdges()
_axes.SetBounds (min(_x)-_maxNorm, max(_x)+_maxNor m, min(_y)-_maxNorm,
max(_y)+_maxNor m, min(_z)-_maxNorm, max(_z)+_maxNor m)
_axes.SetXLabel ("x")
_axes.SetYLabel ("y")
_axes.SetZLabel ("z")
_axesProp = _axes.GetProper ty()
_axesProp.SetCo lor(0,0,0)
_axesTitleProp = _axes.GetAxisTi tleTextProperty ()
_axesTitleProp. ShallowCopy(_te xtProp)
_axesLabelProp = _axes.GetAxisLa belTextProperty ()
_axesLabelProp. ShallowCopy(_te xtProp)
_axesLabelProp. SetFontSize(8)
_renderer.AddAc tor(_axes)
_renderer.Reset Camera()
_renderer.SetBa ckground(1,1,1)
_renderWindow.R ender()
_iRenderer.Star t()

As you can see from the snipping, there's good reason for me to want to
share data objects around... I know it's ugly, but it got it working
quickly (I'm a scientist, not a proper software developer...) Hopefully you
can also see that the pyvisi script is a _lot_ simpler than what is required
to generate the visualisation.

Ok. I think that's enough for one post. Again, thanks heaps for your
reply, I do appreciate it, and hopefully now you can understand my question
a bit more clearly. I look forward to hearing from you soon.

Regards,

Paul

--
Paul Cochrane
Earth Systems Science Computational Centre
University of Queensland Brisbane Queensland 4072 Australia
E: cochrane at esscc dot uq dot edu dot au

**Tom Anderson** · Nov 3 '05, 08:05 PM

Re: Running autogenerated code in another python instance

On Thu, 3 Nov 2005, Paul Cochrane wrote:
[color=blue]
> On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
>[color=green]
>> On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <cochrane@shake 56.esscc.uq.edu .au> wrote:
>>[color=darkred]
>>> I've got an application that I'm writing that autogenerates python
>>> code which I then execute with exec(). I know that this is not the
>>> best way to run things, and I'm not 100% sure as to what I really
>>> should do. I've had a look through Programming Python and the Python
>>> Cookbook, which have given me ideas, but nothing has gelled yet, so I
>>> thought I'd put the question to the community. But first, let me be a
>>> little more detailed in what I want to do:[/color][/color][/color]

Paul, this is a rather interesting problem. There are two aspects to it,
which i believe are probably separable: getting instructions from the
client to the server, and getting data back from the server to the client.
The former is more complex, i think, and what's attracted the attention so
far.

The first thing i'd say is that, while eval/exec is definitely a code
smell, that doesn't mean it's never the right solution. If you need to be
able to express complex things, python code might well be the best way to
do it, and the best way to evaluate python code is eval/exec.
[color=blue][color=green]
>> It's a little hard to tell without knowing more about your user input
>> (command language?) syntax that is translated to or feeds the process
>> that "autogenera tes python code".[/color]
>
> It's basically just a command language I guess.[/color]

Hang on - the stuff that the user writes is what you're calling "pyvisi
code", is that right? That doesn't look like 'just a command language',
that looks like python, using a library you've written. Or is there
another language, the "just a command language", on top of that?

And what you call "vtk-python code" - this is python again, but using the
renderer's native library, right?

And you generate the vtk-python from the pyvisi-python by executing the
pyvisi-python, there being (pluggable renderer-specific) logic in the guts
of your pyvisi classes to emit the vtk-python code, right? You're not
parsing anything?
[color=blue][color=green]
>> There are lots of easy things you could do without generating and exec-ing
>> python code per se.[/color]
>
> I'd love to know of other options. I like the idea of generating the
> code one would have to write for a particular renderer so that if the
> user wanted to, they could use the autogenerated code to form the basis
> of a specific visualisation script they could then hack themselves.[/color]

If you want vtk-python code as an intermediate, i think you're stuck with
eval/exec [1].
[color=blue]
> One of the main ideas of the module is to distill the common visualisation
> tasks down to a simple set of commands, and then let the interface work out
> how to actually implement that.[/color]

Okay. There's a classic design pattern called Interpreter that applies
here. This is one of the more complex patterns, and one that's rather
poorly explained in the Gang of Four book, so it's not well-known.

Basically, the idea is that you provide classes which make it possible for
a program to build structures encoding a series of instructions -
essentially, you define a language whose concrete syntax is objects, not
text - then you write code which takes such structures and carries out the
instructions encoded in them - an interpreter, in other words.

For example, here's a very simple example for doing basic arithmetic:

# the language

class expression(obje ct):
pass

class constant(expres sion):
def __init__(self, value):
self.value = value

class unary(expressio n):
def __init__(self, op, arg):
self.op = op
self.arg = arg

class binary(expressi on):
def __init__(self, op, arg_l, arg_r):
self.op = op
self.arg_l = arg_l
self.arg_r = arg_r

# the interpreter

UNARY_OPS = {
"-": lambda x: -x,
"|": lambda x: abs(x) # apologies for abnormal syntax
}

BINARY_OPS = {
"+": lambda l, r: l + r,
"-": lambda l, r: l - r,
"*": lambda l, r: l * r,
"/": lambda l, r: l / r,
}

def evaluate(expr):
if isinstance(expr , constant):
return expr.value
elif isinstance(expr , unary):
op = UNARY_OPS[expr.op]
arg = evaluate(expr.a rg)
return op(arg)
elif isinstance(expr , binary):
op = BINARY_OPS[expr.op]
arg_l = evaluate(expr.a rg_l)
arg_r = evaluate(expr.a rg_r)
return op(arg_l, arg_r)
else:
raise Exception, "unknown expression type: " + str(type(expr))

# a quick demo

expr = binary("-",
binary("*",
constant(2.0),
constant(3.0)),
unary("-",
binary("/",
constant(4.0),
constant(5.0))) )

print evaluate(expr)

This is by no means a useful or well-designed bit of code, and there are
several things that could have been done differently (bare vs wrapped
constants, operations defined by a symbol vs expression subtypes for each
operation, etc), but i hope it gets the idea across - representing a
language using an object graph, which lets you write programs that can
speak that language.

Your code is already doing something a bit like this - you build scene
graphs, then call render on them to get them to do something. Instead of
that, you'd pass the whole scene to a renderer object, which would do the
rendering (directly, rather than by generating code). The point is that
the renderer could be in another process, provided you have a way to move
the scene graph from one process to another - the pickle module, for
example, or a custom serialisation format if you feel like reinventing the
wheel.

An approach like this has a natural solution to your second problem, too -
the evaluator function can return objects, which again can just be pickled
and sent over the network.

tom

[1] Okay, so there is a way to do this without ever actually creating
python code. You're not going to like this.

You need to apply the interpreter pattern to python itself. Well, a
simplified subset of it. Looking at your generated vtk-python code, you
basically do the following things:

- call methods with variables and literals as arguments
- throwing away the result
- or keeping it in a variable
- do for loops over ranges of integers

To make things a bit simpler, i'm going to add:

- getting attributes
- subscripting arrays
- return a value

You also need to do some arithmetic, i think; i leave that as an exercise
for the reader.

So we need a language like:

class statement(objec t):
pass

class invoke(statemen t):
def __init__(self, var, target, method, args):
self.var = var # name of variable for result; None to throw away
self.target = target
self.method = method
self.args = args

class get(statement):
def __init__(self, var, target, field):
self.var = var
self.target = target
self.field = field

class subscript(state ment):
def __init__(self, var, target, index):
self.var = var
self.target = target
self.index = index

class forloop(stateme nt):
def __init__(self, var, limit, body):
self.var = var
self.limit = limit
self.body = body # tuple of statements

def return_(stateme nt):
def __init__(self, value):
self.value = value

With which we can write a script like:

vtk_script = [
invoke("_plot", "vtk", "vtkXYPlotActor ", ()),
invoke("_render er", "vtk", "vtkRendere r", ()),
invoke("_render Window", "vtk", "vtkRenderWindo w", ()),
invoke(None, "_renderWindow" , "AddRendere r", ("_renderer" )),
invoke(None, "_renderWindow" , "SetSize", (640, 480)),
invoke(None, "_renderer" , "SetBackground" , (1, 1, 1)),
invoke("_x", None, "array", ([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0],)),
get("vtk_vtkDat aArray", "vtk", "vtkDataArray") ,
get("vtk_VTK_FL OAT", "vtk", "VTK_FLOAT" ),
invoke("_xData" , "vtk_vtkDataArr ay", "CreateDataArra y", ("vtk_VTK_FLOAT ",)),
invoke("_x_len" , None, "len", ("_x",)),
invoke(None, "_xData", "SetNumberOfTup les", ("_x_len",)) ,
invoke("_y0", None, "array", ([0.0, 1.0, 4.0, 9.0, 16.0, 25.0, 36.0, 49.0, 64.0, 81.0],)),
invoke("_y0Data ", "vtk_vtkDataArr ay", "CreateDataArra y", ("vtk_VTK_FLOAT ",)),
invoke("_y0_len ", None, "len", ("_y0",)),
invoke(None, "_y0Data", "SetNumberOfTup les", ("_y0_len",) ),
forloop("i", "_x_len", (
subscript("_x_i ", "_x", "i"),
invoke(None, "_xData", "SetTuple1" , ("i", "_x_i")),)) ,
# etc
]

Or rather, your pyvisi classes can generate structures like this, exactly
as they currently generate code.

Code to convert this into python source is trivial, so i'll gloss over
that. The interpreter looks like this:

def isiterable(x):
return hasattr(x, "__iter__")

def execute(script, vars=None):
if (vars == None):
vars = {}
# initialise variables with 'vtk' and anything else you need
def decode_arg(arg) :
if (isinstance(arg , str)):
return vars[arg]
elif (isiterable(arg )):
return map(decode_arg, arg)
else:
return arg
for stmt in script:
if (isinstance(stm t, invoke)):
target = vars[stmt.target]
method = getattr(target, stmt.method)
args = decode_arg(stmt .args)
result = method(*args)
if (stmt.var != None):
vars[stmt.var] = result
elif (isinstance(stm t, get)):
target = vars[stmt.target]
vars[stmt.var] = getattr(target, stmt.field)
elif (isinstance(stm t, subscript)):
target = vars[stmt.target]
vars[stmt.var] = getattr(target, decode_arg(stmt .index))
elif (isinstance(stm t, forloop)):
var = stmt.var
limit = decode_arg(stmt .limit)
body = stmt.body
for i in range(limit):
vars[var] = i
execute(body, vars)
elif (isinstance(stm t, return_)):
return vars[stmt.value]
# nb won't work from inside a for loop!
# you can use an exception to handle returns properly

Note that i haven't tested this, i've just written it off the top of my
head, so it probably won't work, but maybe you get the idea.

To be honest, i'd go with exec.

--
Fitter, Happier, More Productive.

**Bengt Richter** · Nov 7 '05, 12:05 AM

Re: Running autogenerated code in another python instance

On Thu, 03 Nov 2005 14:23:53 +1000, Paul Cochrane <cochrane@esscc .uq.edu.au> wrote:
[color=blue]
>On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
>[color=green]
>> On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <cochrane@shake 56.esscc.uq.edu .au> wrote:
>>[color=darkred]
>>>Hi all,
>>>
>>>I've got an application that I'm writing that autogenerates python code
>>>which I then execute with exec(). I know that this is not the best way to
>>>run things, and I'm not 100% sure as to what I really should do. I've had a
>>>look through Programming Python and the Python Cookbook, which have given me
>>>ideas, but nothing has gelled yet, so I thought I'd put the question to the
>>>community. But first, let me be a little more detailed in what I want to
>>>do:
>>>[/color][/color]
>
>Bengt,
>
>Thanks for your reply!
>[color=green]
>> It's a little hard to tell without knowing more about your
>> user input (command language?) syntax that is translated to
>> or feeds the process that "autogenera tes python code".[/color]
>Ok, I'll try and clarify things as much as I can.
>[/color]
[...snip great reply...]

I owe you another reply, but I started and I couldn't spend the time to
do it justice. But in the meanwhile, I would suggest thinking about how
the MVC (model-view-controller) concept might help you factor things.
ISTM you are already part way there. See

MVC - Wikipedia

http://en.wikipedia.org/wiki/MVC

for some infos. To that I'd add that if you want a top level interactive visualization tool,
you might want to look at it like a kind of IDE, where you might want a few workspace/project kind
of top level commands to help manage what you would otherwise do by way of manually creating
directory subtrees for various things etc. Anyway, if you want to use template code and edit
it and then compile and run it, you could select templates and make working copies automatically
and invoke some favorite editor on it from the top level command interpreter. Having these
pieces consistently arranged in projects/workspaces and shared template spaces etc. would
make it a single command to create a fresh space and not worry about colliding with something
you did just to show someone a little demo, etc. This is not the central topic, but good useability
is nice ;-)

That way if you just wanted to re-run something, you'd just select it and skip calling the
editor. Or if a step was to generate data, you could either create the data source program
by several steps or possibly just go on to define or invoke a visualization step with
a particular renderer and/or output, knowing that the data source was already set up in a
standard way. You could also consider borrowing unix piping/redirection concepts
for some command syntax, for composition of standard interface actions (not to mention
invoking the real thing in a subprocess when appropriate). Just free-associating here ;-)

Anyway, gotta go for now, sorry.

Regards,
Bengt Richter

**Paul Cochrane** · Nov 7 '05, 01:15 AM

Re: Running autogenerated code in another python instance

On Thu, 03 Nov 2005 19:56:48 +0000, Tom Anderson wrote:
[color=blue]
> On Thu, 3 Nov 2005, Paul Cochrane wrote:
>[color=green]
>> On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
>>[color=darkred]
>>> On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <cochrane@shake 56.esscc.uq.edu .au> wrote:
>>>
>>>> I've got an application that I'm writing that autogenerates python
>>>> code which I then execute with exec(). I know that this is not the
>>>> best way to run things, and I'm not 100% sure as to what I really
>>>> should do. I've had a look through Programming Python and the Python
>>>> Cookbook, which have given me ideas, but nothing has gelled yet, so I
>>>> thought I'd put the question to the community. But first, let me be a
>>>> little more detailed in what I want to do:[/color][/color][/color]
[color=blue]
> Paul, this is a rather interesting problem. There are two aspects to it,
> which i believe are probably separable: getting instructions from the
> client to the server, and getting data back from the server to the client.
> The former is more complex, i think, and what's attracted the attention so
> far.[/color]
Tom, thanks heaps for your reply!
[color=blue]
> The first thing i'd say is that, while eval/exec is definitely a code
> smell, that doesn't mean it's never the right solution. If you need to be
> able to express complex things, python code might well be the best way to
> do it, and the best way to evaluate python code is eval/exec.[/color]
After looking at the problem a bit more I've come up with a simpler
solution to the problem I initially posted; and it involves exec. I've
realised that what I've been doing wrong is to compile the code first
before I exec it. If I just exec the generated code (within a predefined
namespace) then pyvisi does all of the things I want it to do without
needing separate processes for generating and rendering the code. This
has also solved my data passing problem. I now just need to pass a
reference to the data into the namespace where I'm running the
generated code and it all works really nicely. Basically the problem had
been staring me in the face for ages and I just hadn't seen it. (duh!)
[color=blue][color=green][color=darkred]
>>> It's a little hard to tell without knowing more about your user input
>>> (command language?) syntax that is translated to or feeds the process
>>> that "autogenera tes python code".[/color]
>>
>> It's basically just a command language I guess.[/color]
>
> Hang on - the stuff that the user writes is what you're calling "pyvisi
> code", is that right? That doesn't look like 'just a command language',
> that looks like python, using a library you've written. Or is there
> another language, the "just a command language", on top of that?[/color]
You're right, it isn't a command language (I looked up the cmd module in
Python in a Nutshell and then realised that a large part of my reply was
in the "I guess" part). It is just python code that is being run.[color=blue]
>
> And what you call "vtk-python code" - this is python again, but using
> the renderer's native library, right?[/color]
That's correct.[color=blue]
>
> And you generate the vtk-python from the pyvisi-python by executing the
> pyvisi-python, there being (pluggable renderer-specific) logic in the
> guts of your pyvisi classes to emit the vtk-python code, right? You're
> not parsing anything?[/color]
No parsing going on. Just a translation of high-level ideas into the low
level of the renderer underneath.[color=blue]
>[color=green][color=darkred]
>>> There are lots of easy things you could do without generating and
>>> exec-ing python code per se.[/color]
>>
>> I'd love to know of other options. I like the idea of generating the
>> code one would have to write for a particular renderer so that if the
>> user wanted to, they could use the autogenerated code to form the basis
>> of a specific visualisation script they could then hack themselves.[/color]
>
> If you want vtk-python code as an intermediate, i think you're stuck
> with eval/exec [1].[/color]
So do I. And as I said above, it seems to have simplified my problem
significantly. I've also managed to speed the code up as well!

Thanks heaps for the rest of your comments and suggestions. They've been
really helpful for me to see alternative and probably cleaner ways of
doing what I want to do.
[color=blue]
> To be honest, i'd go with exec.[/color]
:-)

Again, thanks for your help, I really do appreciate it.

Paul

**Paul Cochrane** · Nov 7 '05, 01:35 AM

Re: Running autogenerated code in another python instance

On Mon, 07 Nov 2005 00:01:29 +0000, Bengt Richter wrote:
[color=blue]
> On Thu, 03 Nov 2005 14:23:53 +1000, Paul Cochrane <cochrane@esscc .uq.edu.au> wrote:
>[color=green]
>>On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
>>[color=darkred]
>>> On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <cochrane@shake 56.esscc.uq.edu .au> wrote:
>>>
>>>>Hi all,
>>>>
>>>>I've got an application that I'm writing that autogenerates python code
>>>>which I then execute with exec(). I know that this is not the best way to
>>>>run things, and I'm not 100% sure as to what I really should do. I've had a
>>>>look through Programming Python and the Python Cookbook, which have given me
>>>>ideas, but nothing has gelled yet, so I thought I'd put the question to the
>>>>community . But first, let me be a little more detailed in what I want to
>>>>do:
>>>>[/color]
>>
>>Bengt,
>>
>>Thanks for your reply!
>>[color=darkred]
>>> It's a little hard to tell without knowing more about your
>>> user input (command language?) syntax that is translated to
>>> or feeds the process that "autogenera tes python code".[/color]
>>Ok, I'll try and clarify things as much as I can.
>>[/color]
> [...snip great reply...][/color]
Thanks :-)[color=blue]
>
> I owe you another reply, but I started and I couldn't spend the time to
> do it justice. But in the meanwhile, I would suggest thinking about how
> the MVC (model-view-controller) concept might help you factor things.
> ISTM you are already part way there. See
> http://en.wikipedia.org/wiki/MVC[/color]
Thanks for the tip.
[color=blue]
> for some infos. To that I'd add that if you want a top level interactive visualization tool,
> you might want to look at it like a kind of IDE, where you might want a few workspace/project kind
> of top level commands to help manage what you would otherwise do by way of manually creating
> directory subtrees for various things etc. Anyway, if you want to use template code and edit
> it and then compile and run it, you could select templates and make working copies automatically
> and invoke some favorite editor on it from the top level command interpreter. Having these
> pieces consistently arranged in projects/workspaces and shared template spaces etc. would
> make it a single command to create a fresh space and not worry about colliding with something
> you did just to show someone a little demo, etc. This is not the central topic, but good useability
> is nice ;-)[/color]
I think this is probably a bit more complex than I really want. All I
want to do is to take the pain out of scripting visualisation of large and
complex data sets. My code will eventually run inside a batch job, or
behind a web-based GRID interface thing (still being developed by other
people in my group). The interactive part of vtk is just a
handy side effect; allowing users to rotate objects within a window and
see what's on the other side etc is really nice, but not necessarily the
main aim. Nevertheless, I've found out what I've been doing wrong all
along: I've been compiling the code objects before exec-ing them. By just
running exec on the generated code I can run the code generation and
rendering within the one process (since not compiling beforehand allows me
to maintain state), and I can pass the data around as the objects
themselves and not this convoluted convert to string, convert back to
number process I had going. It's also now a lot faster (which is always a
bonus)!
[color=blue]
> That way if you just wanted to re-run something, you'd just select it and skip calling the
> editor. Or if a step was to generate data, you could either create the data source program
> by several steps or possibly just go on to define or invoke a visualization step with
> a particular renderer and/or output, knowing that the data source was already set up in a
> standard way. You could also consider borrowing unix piping/redirection concepts
> for some command syntax, for composition of standard interface actions (not to mention
> invoking the real thing in a subprocess when appropriate). Just free-associating here ;-)[/color]
These are great ideas, but probably a bit more than I can handle atm :-)
[color=blue]
> Anyway, gotta go for now, sorry.[/color]
No worries! I really appreciate your feedback.

Paul

Running autogenerated code in another python instance

Running autogenerated code in another python instance

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment