Python code written in 1998, how to improve/change it?

[Hans Georg Krauthaeuser]

Re: [OT] no goto (WAS: Python code written in 1998...)

Steven Bethard schrieb:[color=blue]
> Carl Cerecke wrote:
>[color=green]
>> Python has no goto.[/color]
>
>
> Not in the standard library. You have to download the module:
> http://www.entrian.com/goto/
>
> ;)
>
> STeVe[/color]
This remerbers me to VATICAL, a famous programming language from the 80s.

http://www.uni-weimar.de/~mildenbe/spass/vatical/vatical.html

It's in German only and I'm not aware of a English translation.

Please be aware that content of that page is not political correct and
may offend your religious sensibilities.

Hans Georg

[Bengt Richter]

Re: Python code written in 1998, how to improve/change it?

On Thu, 19 Jan 2006 23:16:57 -0500, Peter Hansen <peter@engcorp. com> wrote:
[color=blue]
>Carl Cerecke wrote:[color=green]
>> Carl Cerecke wrote:[color=darkred]
>>>Ah. Well, my post suggested, as one option, the callables call
>>>each other directly.[/color]
>>
>> Doh! No I didn't. And they shouldn't. Otherwise the call stack
>> gets out of hand. But I did suggest that each callable representing a
>> state set a global variable, just before it returns, to the callable
>> representing the next state to be called. Still no map required. Just a
>> while loop. In any case, the function call/return is wasted cycles.[/color]
>
>I believe the more modern approach to this is to use generators in some
>way, yield each other as the next state. This way you avoid all almost
>all the function call overhead (the part that takes significant time,
>which is setting up the stack frame) and don't have to resort to
>bytecode hacks for better performance.
>
>Of course, if you have a state machine with many small states each doing
>a tiny bit of processing and you're still concerned over performance,
>you probably should be looking into Pysco or Pyrex and avoid making your
>code really unreadable.
>[/color]
How about something like
[color=blue][color=green][color=darkred]
>>> actions = dict([/color][/color][/color]
... a=compile('prin t "A"; state="b"','',' exec'),
... b=compile('prin t "B"; state="c"','',' exec'),
... c=compile('prin t "C"; state=None','', 'exec')
... )[color=blue][color=green][color=darkred]
>>> state = 'a'
>>> while state: eval(actions[state])[/color][/color][/color]
...
A
B
C

Regards,
Bengt Richter

[Steven D'Aprano]

Re: Python code written in 1998, how to improve/change it?

On Fri, 20 Jan 2006 10:27:58 +1300, Carl Cerecke wrote:
[color=blue]
> We want a goto.
>
> Unfortunately, this is about the only problem I can think of where gotos
> are useful. And for reasons well explained elsewhere, gotos are Not Good.
>
> I've solved this in a very efficient, but rather unpythonic way.[/color]

There is a module that does both GOTO and COMEFROM.

http://www.entrian.com/goto/

PLEASE don't use it in real code.



--
Steven.

[Carl Cerecke]

Re: Python code written in 1998, how to improve/change it?

Bengt Richter wrote:[color=blue]
> On Thu, 19 Jan 2006 23:16:57 -0500, Peter Hansen <peter@engcorp. com> wrote:[/color]
[color=blue]
> How about something like
>[color=green][color=darkred]
> >>> actions = dict([/color][/color]
> ... a=compile('prin t "A"; state="b"','',' exec'),
> ... b=compile('prin t "B"; state="c"','',' exec'),
> ... c=compile('prin t "C"; state=None','', 'exec')
> ... )[color=green][color=darkred]
> >>> state = 'a'
> >>> while state: eval(actions[state])[/color][/color]
> ...
> A
> B
> C[/color]

Good idea. But we can eliminate the dictionary lookup:

a1 = compile('print "A"; state=b1','','e xec')
b1 = compile('print "B"; state=c1','','e xec')
c1 = compile('print "C"; state=None','', 'exec')

state = a1
while state:
eval(state)


Cheers,
Carl

[Petr Jakes]

Re: Python code written in 1998, how to improve/change it?

Sorry, I can't get in. Can you please show me, how to use your approach
on the simple push/push ON/OFF button for example please?

PS: seriously it is not a homework :) and I feel it like a shame I am
asking such a simple questions :(

States: ON, OFF
Transition event: "push", "lift"

transition diagram:
=============== ==========

___ lift
| |
_V___|__
,->| ON |__
| |________| |
lift | | push
| ________ |
'--| OFF |<-'
|________|
^ |
|___|push

[Petr Jakes]

Re: Python code written in 1998, how to improve/change it?

Sorry for the typo in my previous posting. Of course it has to be: ....
simple liftt/push ON/OFF button....

[Carl Cerecke]

Re: Python code written in 1998, how to improve/change it?

Petr Jakes wrote:[color=blue]
> Sorry, I can't get in. Can you please show me, how to use your approach
> on the simple push/push ON/OFF button for example please?
>
> PS: seriously it is not a homework :) and I feel it like a shame I am
> asking such a simple questions :(
>
> States: ON, OFF
> Transition event: "push", "lift"
>
> transition diagram:
> =============== ==========
>
> ___ lift
> | |
> _V___|__
> ,->| ON |__
> | |________| |
> lift | | push
> | ________ |
> '--| OFF |<-'
> |________|
> ^ |
> |___|push
>[/color]

As a starting point, how about:

l = 'lift'
p = 'push'

action_sequence = [l,p,p,l,l,p,l,p ,None]
next_action = iter(action_seq uence).next

s_on = compile('''
print 'on'
action = next_action()
if action == 'lift':
state = s_on
elif action == 'push':
state = s_off
else:
state = None
''','','exec')

s_off = compile('''
print 'off'
action = next_action()
if action == 'lift':
state = s_on
elif action == 'push':
state = s_off
else:
state = None
''','','exec')


state = s_on # start state
while state:
eval(state)



Cheers,
Carl

[Bengt Richter]

Re: Python code written in 1998, how to improve/change it?

On Mon, 23 Jan 2006 08:53:59 +1300, Carl Cerecke <cdc@maxnet.co. nz> wrote:
[color=blue]
>Bengt Richter wrote:[color=green]
>> On Thu, 19 Jan 2006 23:16:57 -0500, Peter Hansen <peter@engcorp. com> wrote:[/color]
>[color=green]
>> How about something like
>>[color=darkred]
>> >>> actions = dict([/color]
>> ... a=compile('prin t "A"; state="b"','',' exec'),
>> ... b=compile('prin t "B"; state="c"','',' exec'),
>> ... c=compile('prin t "C"; state=None','', 'exec')
>> ... )[color=darkred]
>> >>> state = 'a'
>> >>> while state: eval(actions[state])[/color]
>> ...
>> A
>> B
>> C[/color]
>
>Good idea. But we can eliminate the dictionary lookup:
>
>a1 = compile('print "A"; state=b1','','e xec')
>b1 = compile('print "B"; state=c1','','e xec')
>c1 = compile('print "C"; state=None','', 'exec')
>
>state = a1
>while state:
> eval(state)
>[/color]
Cool. But unfortunately, neither version works inside a function's local namespace.
Using exec instead of eval seems to do it in either context though.
Now, how can we get optimized code (i.e., LOAD_FAST vs LOAD_NAME etc in a1 etc)
without byte code hackery?

Regards,
Bengt Richter

[Wolfgang Keller]

Re: Python code written in 1998, how to improve/change it?

On Fri, 20 Jan 2006 05:16:57 +0100, Peter Hansen wrote
(in article <mailman.821.11 37730663.27775. python-list@python.org >):
[color=blue]
> I believe the more modern approach to this is to use generators in some
> way, yield each other as the next state. This way you avoid all almost
> all the function call overhead (the part that takes significant time,
> which is setting up the stack frame) and don't have to resort to
> bytecode hacks for better performance.[/color]

Dumb question from a endless newbie scripting dilettant:

Do you have a reference to a cookbook example for this method?

Sidequestion: As I understand it, as a general rule generators are more
efficient than functions in any case where the function is called several
times, right? So basically if I want to write a long-running program in
Python, it would make sense to code all functions that are likely to be
called more than once as generators...

TIA,

Sincerely,

Wolfgang Keller

[Carl Cerecke]

Weird generator id() behaviour (was Re: Python code written in 1998,how to improve/change it?)

Wolfgang Keller wrote:[color=blue]
> On Fri, 20 Jan 2006 05:16:57 +0100, Peter Hansen wrote
> (in article <mailman.821.11 37730663.27775. python-list@python.org >):
>
>[color=green]
>>I believe the more modern approach to this is to use generators in some
>>way, yield each other as the next state. This way you avoid all almost
>>all the function call overhead (the part that takes significant time,
>>which is setting up the stack frame) and don't have to resort to
>>bytecode hacks for better performance.[/color]
>
>
> Dumb question from a endless newbie scripting dilettant:
>
> Do you have a reference to a cookbook example for this method?[/color]

It turns out that generators are more efficient than the eval function
excuting bits of compiled code. About 20-25% faster.

However, the generators exhibit some weird behaviour.

The included code shows an example using a compile/eval FSM method, and
a generator method. Note, in particular, the pattern of ids of the
generator objects. I had expected some complications in the ids, but not
a repeatable sequence of length 3 after the first generator object.

What is going on? Anybody?

#!/usr/bin/env python

import time

s_on = compile('''
print 'on'
action = next_action()
if action == 'lift':
state = s_on
elif action == 'push':
state = s_off
else:
state = None
''','','exec')

s_off = compile('''
print 'off'
action = next_action()
if action == 'lift':
state = s_on
elif action == 'push':
state = s_off
else:
state = None
''','','exec')


def g_on():

print "on"
action = next_action()
if action == 'lift':
yield g_on()
elif action == 'push':
yield g_off()
else:
yield None

def g_off():

print "off"
action = next_action()
if action == 'lift':
yield g_on()
elif action == 'push':
yield g_off()
else:
yield None


def actions(n):
import random
for i in range(n-1):
yield random.choice(['lift','push'])
yield None

#r = 1000000
r = 10
next_action = actions(r).next

#a = time.clock()
while next_action():
pass
#z = time.clock()
#print "action generator:",z-a
next_action = actions(r).next
#print "---"
state = s_on # start state
#a = time.clock()
while state:
eval(state)
#z = time.clock()
#print z-a
print "---"

next_action = actions(r).next
s_g_on = g_on()
s_g_off = g_off()
state = s_g_on
#a = time.clock()
while state:
print id(state)
state = state.next()
#z = time.clock()
#print z-a


#Cheers,
#Carl.

[Wolfgang Keller]

Re: Weird generator id() behaviour (was Re: Python code written in 1998, how to improve/change it?)

On Tue, 24 Jan 2006 21:19:26 +0100, Carl Cerecke wrote
(in article <43d68bd0@usene t01.boi.hp.com> ):
[color=blue]
> def g_on():
>
> print "on"
> action = next_action()
> if action == 'lift':
> yield g_on()
> elif action == 'push':
> yield g_off()
> else:
> yield None
>
> def g_off():
>
> print "off"
> action = next_action()
> if action == 'lift':
> yield g_on()
> elif action == 'push':
> yield g_off()
> else:
> yield None[/color]

Amazing.

Executable pseudo-code, really. :-)

And that's even (run-time) efficient?

Tanks a lot,

Sincerely,

Wolfgang Keller

[skip@pobox.com]

Re: Python code written in 1998, how to improve/change it?


Wolfgang> So basically if I want to write a long-running program in
Wolfgang> Python, it would make sense to code all functions that are
Wolfgang> likely to be called more than once as generators...

If they need to resume their calculations from where they left off after the
last yield.

Skip

[Fredrik Lundh]

Re: Weird generator id() behaviour (was Re: Python code written in1998, how to improve/change it?)

Carl Cerecke wrote:
[color=blue]
> It turns out that generators are more efficient than the eval function
> excuting bits of compiled code. About 20-25% faster.[/color]

why are you using generators to return things from a function, when
you can just return the things ?

def f_on():
print "on"
action = next_action()
if action == 'lift':
return f_on
elif action == 'push':
return f_off

def f_off():
print "off"
action = next_action()
if action == 'lift':
return f_on
elif action == 'push':
return f_off

state = f_on
while state:
state = state()

</F>

[Carl Cerecke]

Re: Weird generator id() behaviour (was Re: Python code written in1998,how to improve/change it?)

Fredrik Lundh wrote:[color=blue]
> Carl Cerecke wrote:
>
>[color=green]
>>It turns out that generators are more efficient than the eval function
>>excuting bits of compiled code. About 20-25% faster.[/color]
>
>
> why are you using generators to return things from a function, when
> you can just return the things ?[/color]

Trying to find the fastest way to implement finite state machine.
The included file times 4 different ways, with functions the fastest.

The reason, I think, for the unusual sequence if id()s of the generators
- see grandparent post - (and the reason for their poor performance
compared to functions), is because the reference to the generator is
being lost, then another generator is being created with the same id.
Properly done, I would expect generators to out perform functions.

These are the numbers I get on a P3 600:
$ ./foo.py
action generator overhead: 13.25
---
exec: 8.7
---
eval: 10.09
---
generators: 6.68
---
functions: 3.37


#!/usr/bin/env python

import time

s_on = compile('''
#print 'on'
action = next_action()
if action == 'lift':
state = s_on
elif action == 'push':
state = s_off
else:
state = None
''','','exec')

s_off = compile('''
#print 'off'
action = next_action()
if action == 'lift':
state = s_on
elif action == 'push':
state = s_off
else:
state = None
''','','exec')

def f_on():
action = next_action()
if action == 'lift':
return f_on
elif action == 'push':
return f_off
else:
return None

def f_off():
action = next_action()
if action == 'lift':
return f_on
elif action == 'push':
return f_off
else:
return None


def g_on():

#print "on"
action = next_action()
if action == 'lift':
yield g_on()
elif action == 'push':
yield g_off()
else:
yield None

def g_off():

#print "off"
action = next_action()
if action == 'lift':
yield g_on()
elif action == 'push':
yield g_off()
else:
yield None


def actions(n):
import random
for i in range(n-1):
yield random.choice(['lift','push'])
yield None

r = 1000000
#r = 10
next_action = actions(r).next

a = time.clock()
while next_action():
pass
z = time.clock()
print "action generator overhead:",z-a
common = z-a

print "---"
next_action = actions(r).next
state = s_on # start state
a = time.clock()
while state:
exec state
z = time.clock()
print "exec:",z-a-common

print "---"
next_action = actions(r).next
state = s_on # start state
a = time.clock()
while state:
eval(state)
z = time.clock()
print "eval:",z-a-common

print "---"
next_action = actions(r).next
s_g_on = g_on()
s_g_off = g_off()
state = s_g_on
a = time.clock()
while state:
#print id(state)
state = state.next()
z = time.clock()
print "generators :",z-a-common

print "---"
next_action = actions(r).next
state = f_on
a = time.clock()
while state:
state = state()
z = time.clock()
print "functions: ",z-a-common

[Carl Cerecke]

Re: Weird generator id() behaviour (was Re: Python code written in1998,howto improve/change it?)

Carl Cerecke wrote:[color=blue]
> Fredrik Lundh wrote:
>[color=green]
>> Carl Cerecke wrote:
>>
>>[color=darkred]
>>> It turns out that generators are more efficient than the eval function
>>> excuting bits of compiled code. About 20-25% faster.[/color]
>>
>>
>>
>> why are you using generators to return things from a function, when
>> you can just return the things ?[/color]
>
>
> Trying to find the fastest way to implement finite state machine.
> The included file times 4 different ways, with functions the fastest.
>
> The reason, I think, for the unusual sequence if id()s of the generators
> - see grandparent post - (and the reason for their poor performance
> compared to functions), is because the reference to the generator is
> being lost, then another generator is being created with the same id.
> Properly done, I would expect generators to out perform functions.[/color]

Generator FSM done properly (well, better anyway). They are still almost
twice as slow as plain function calls, though.

def g_on():

while 1:
action = next_action()
if action == 'lift':
yield s_g_on
elif action == 'push':
yield s_g_off
else:
break
yield None

def g_off():

while 1:
action = next_action()
if action == 'lift':
yield s_g_on
elif action == 'push':
yield s_g_off
else:
break
yield None

def actions(n):
import random
for i in range(n-1):
yield random.choice(['lift','push'])
yield None

r = 1000000
#r = 10
next_action = actions(r).next
s_g_on = g_on()
s_g_off = g_off()
state = s_g_on

while state:
state = state.next()
z = time.clock()