Too Many if Statements?

[Kent Johnson]

Re: Too Many if Statements?

slogging_away wrote:[color=blue]
> bruno at modulix wrote:[color=green]
>>Why storing error messages ? Why don't you just write'em out (be it to
>>stdout or to a file) ?[/color]
>
>
> I guess I could do that, (write them to a file as they are discovered).
> Right now the error messages are stored in the array and then the the
> array is scanned via, a for loop and the error messages are written to
> several files in different formats based on the severity of the errors,
> (on a per device basis, a per severity basis, etc.). This keeps the
> write statements to a minimum and in a central location of the script
> instead of having several statements for each individual error message
> spread throughout the script, (three write statements per error message
> at over 500 error messages would be a significant change).[/color]

Sounds like you might like the logging module. A single log entry can be
written to multiple destinations based on level or source of the entry.
The distribution of log entries is controlled by the logging
configuration which can be stored in a text file if you like.

Kent

[slogging_away]

Re: Too Many if Statements?

Magnus Lycka wrote:
[color=blue]
> What happens if you run it from the command line instead
> of IDLE? After all, it might be some problem in IDLE involved
> here. Even if it doesn't work correctly outside IDLE, I was
> thinking that IDLE might swallow some kind of error message.[/color]

Excellent suggestion, (behold the power of the command line!). I ran
two saved versions of the script that had produced the symptom
originally described. The fist seemed to be caused by too many 'if'
statements, the second by adding another array, but both came up with
the same system error at the command prompt level shown here:

SystemError: com_backpatch: offset too large

This error is not produced with the IDLE console but is seen only when
executing from the command line. I'll search around and see if I can
determine what this means and a possible fix. Thanks for the
suggestion!

[Steven D'Aprano]

Re: Too Many if Statements?

On Fri, 10 Feb 2006 06:50:25 -0800, slogging_away wrote:
[color=blue]
> Excellent suggestion, (behold the power of the command line!). I ran
> two saved versions of the script that had produced the symptom
> originally described. The fist seemed to be caused by too many 'if'
> statements, the second by adding another array, but both came up with
> the same system error at the command prompt level shown here:
>
> SystemError: com_backpatch: offset too large[/color]

Is this a Python error or a shell error?

If it is a Python error, perhaps you would like to post the entire
traceback?



--
Steven.

[Juho Schultz]

Re: Too Many if Statements?

Steven D'Aprano wrote:[color=blue]
> On Fri, 10 Feb 2006 06:50:25 -0800, slogging_away wrote:
>
>[color=green]
>>Excellent suggestion, (behold the power of the command line!). I ran
>>two saved versions of the script that had produced the symptom
>>originally described. The fist seemed to be caused by too many 'if'
>>statements, the second by adding another array, but both came up with
>>the same system error at the command prompt level shown here:
>>
>>SystemError : com_backpatch: offset too large[/color]
>
>
> Is this a Python error or a shell error?
>
> If it is a Python error, perhaps you would like to post the entire
> traceback?
>
>
>[/color]

I would believe CamelCaseErrorM essages are produced by Python.

The message is exactly the same I reported with the 2500 elifs. I fooled
around a bit with this, and it seems that also for/while blocks
containing more than ~4860 lines give this error.

slogging_away claims his script is about 4900 lines, most of that in a
for loop, so my bet is he has trouble with the same bug.

[Terry Reedy]

Re: Too Many if Statements?


"slogging_a way" <hinnc@yahoo.co m> wrote in message
news:1139583025 .428875.225500@ g14g2000cwa.goo glegroups.com.. .[color=blue]
> SystemError: com_backpatch: offset too large[/color]

This message is generated in the backpatch function in Python/compile.c in
the source tree. (See below: sorry, tab indents did not survive cut and
paste operation.) As the comment says, and the code shows, it patches in a
2-byte jump offset. The error is raised if the offset is not 0 after
beinging divided by 256 twice (ie, by 65536). This code is called from any
function that compiles a construct that potentially jumps: for, while,
if/elif, and short-circuiting logic expressions.

To me, this message indicates not a bug but a mismatch between code demand
and interpreter capability. The solution is to reduce the demand. In the
present case, changing

for line in file:
<extra-long body>

to

def checkline(line) :
<extra-long body>
for line in file:
checkline(line)

might be sufficient. If not, break up the <extra-lone body> into multiple
pieces.

Terry Jan Reedy

static void
com_backpatch(s truct compiling *c, int anchor)
{
unsigned char *code = (unsigned char *) PyString_AS_STR ING(c->c_code);
int target = c->c_nexti;
int dist;
int prev;
for (;;) {
/* Make the JUMP instruction at anchor point to target */
prev = code[anchor] + (code[anchor+1] << 8);
dist = target - (anchor+2);
code[anchor] = dist & 0xff;
dist >>= 8;
code[anchor+1] = dist;
dist >>= 8;
if (dist) {
com_error(c, PyExc_SystemErr or,
"com_backpa tch: offset too large");
break;
}
if (!prev)
break;
anchor -= prev;
}
}

[Steve Holden]

Re: Too Many if Statements?

slogging_away wrote:[color=blue]
> Magnus Lycka wrote:
>
>[color=green]
>>What happens if you run it from the command line instead
>>of IDLE? After all, it might be some problem in IDLE involved
>>here. Even if it doesn't work correctly outside IDLE, I was
>>thinking that IDLE might swallow some kind of error message.[/color]
>
>
> Excellent suggestion, (behold the power of the command line!). I ran
> two saved versions of the script that had produced the symptom
> originally described. The fist seemed to be caused by too many 'if'
> statements, the second by adding another array, but both came up with
> the same system error at the command prompt level shown here:
>
> SystemError: com_backpatch: offset too large
>
> This error is not produced with the IDLE console but is seen only when
> executing from the command line. I'll search around and see if I can
> determine what this means and a possible fix. Thanks for the
> suggestion!
>[/color]
Now we have seen the real error message it does appear as though this is
an error in the interpreter triggered by an unusually long source
program. See

http://mail.python.org/pipermail/python-list/2004-November/249270.html

for a previous discussion. In that email Diez Roggisch says:

"""I can't believe that a program with 10000 lines only adding strings
can't be written in a more concise way, which would most probably solve
your problem. So I suggest you post some parts of your code so that we
can have a look at it and suggest a solution."""

The same applies here.

Clearly it would be a good idea to remove whatever problem is causing
the error, but in the meantime is there some way your code could be
written more concisely? Or even split up into multiple modules?

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Terry Reedy]

Re: Too Many if Statements?


"Steve Holden" <steve@holdenwe b.com> wrote in message
news:dsid2f$b99 $1@sea.gmane.or g...[color=blue]
> Clearly it would be a good idea to remove whatever problem is causing
> the error,[/color]

The problem (see my post of the com_backpatch code) is writing a compound
statement (here a for loop) with a body so large as to require a jump of
more than 64K bytes in the compiled bytecode (ie, from the test at the top
of the loop to the code that follows after the loop). Until the jump limit
is raised (likely a long wait ;-), the OP must factor some of the code out
of the loop.

Terry Jan Reedy

[slogging_away]

Re: Too Many if Statements?

Now that I know the root cause of the problem, I can write around it as
suggested by Steve Holden and Terry Reedy, (and others). In fact, it
has helped me in a way as I am thinking not in terms of the easiest
solution, (read; the first one that comes to mind), but more effcient
and cleaner ways to write a section of code to accomplish the same
objective. The key was identifying the root cause which was provided by
the error message seen only at the command line level and by
contibutors to this post.

Thanks again for everyone's suggestions and expertise.

[Bengt Richter]

Re: Too Many if Statements?

On Sat, 11 Feb 2006 15:40:49 -0500, "Terry Reedy" <tjreedy@udel.e du> wrote:
[color=blue]
>
>"Steve Holden" <steve@holdenwe b.com> wrote in message
>news:dsid2f$b9 9$1@sea.gmane.o rg...[color=green]
>> Clearly it would be a good idea to remove whatever problem is causing
>> the error,[/color]
>
>The problem (see my post of the com_backpatch code) is writing a compound
>statement (here a for loop) with a body so large as to require a jump of
>more than 64K bytes in the compiled bytecode (ie, from the test at the top
>of the loop to the code that follows after the loop). Until the jump limit
>is raised (likely a long wait ;-), the OP must factor some of the code out
>of the loop.
>[/color]
Easy example:
[color=blue][color=green][color=darkred]
>>> def test(n):[/color][/color][/color]
... while True:
... try: co = compile('if x:\n'+ n*' a=1\n','','exec ')
... except Exception,e: break
... n += 1
... print 'Stopped at n=%s due to %s: %s'%(n, e.__class__.__n ame__,e)
...

get an idea of where to start that:
[color=blue][color=green][color=darkred]
>>> import dis
>>> n=3
>>> dis.dis( compile('if x:\n'+ n*' a=1\n','','exec '))[/color][/color][/color]
1 0 LOAD_NAME 0 (x)
3 JUMP_IF_FALSE 22 (to 28)
6 POP_TOP

2 7 LOAD_CONST 0 (1)
10 STORE_NAME 1 (a)

3 13 LOAD_CONST 0 (1)
16 STORE_NAME 1 (a)

4 19 LOAD_CONST 0 (1)
22 STORE_NAME 1 (a)
25 JUMP_FORWARD 1 (to 29)[color=blue][color=green]
>> 28 POP_TOP
>> 29 LOAD_CONST 1 (None)[/color][/color]
32 RETURN_VALUE[color=blue][color=green][color=darkred]
>>> (2**16-7)/(13-7)[/color][/color][/color]
10921

back off 1
[color=blue][color=green][color=darkred]
>>> test(10920)[/color][/color][/color]
Stopped at n=10922 due to SystemError: com_backpatch: offset too large

Decided to test the exact 65536 jump with code chunks of 16 byte-codes and one
chunk at the end to make 16 with the last JUMP_FORWARD.
[color=blue][color=green][color=darkred]
>>> n=4095
>>> dis.dis( compile('if x:\n'+ n*' a=1+2,4 \n'+' x=0,x','','exec '))[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
SystemError: com_backpatch: offset too large[color=blue][color=green][color=darkred]
>>> n=4094
>>> dis.dis( compile('if x:\n'+ n*' a=1+2,4 \n'+' x=0,x','','exec '))[/color][/color][/color]
1 0 LOAD_NAME 0 (x)
3 JUMP_IF_FALSE 65520 (to 65526)
6 POP_TOP

2 7 LOAD_CONST 0 (1)
10 LOAD_CONST 1 (2)
13 BINARY_ADD
14 LOAD_CONST 2 (4)
17 BUILD_TUPLE 2
20 STORE_NAME 1 (a)

3 23 LOAD_CONST 0 (1)

So the corner case of 2**16 is ok. Believe it or not, I once discovered a compiler
error based on optimizing a 2**16-involving loop condition as if it were 0 and the
loop didn't execute! IIRC, I bumped into it processing an array of records with a stride of
exactly 2**n and it thought it could calculate a 16-bit number of strides for end of loop.
No good for arraysize/stridesize==2** 16 ;-)

If the OP really HAD to, he could always (untested) break

if cond:
too
large
else:
two
large,also
into
if cond:
too
else:
two
if cond:
large
else:
large,also

but that reads gawdawfully. (So, I imagine, does about any code hitting the offset limit ;-)

If it's a matter of too many elifs, the OP could break that more readably, e.g. (untested)

done=True
if cond:
bla
elif c2:
bla 2
...
elif c456:
bla456
else:
done=False

more, done = not done,True
if more and c457:
bla c457
elif c458:
bla458
...
elif c1012:
bla1012
else:
done = False

more, done = not done,True
... etc

But the OP should find another approach I think,
because this is still disgusting code ;-)

Regards,
Bengt Richter

[bruno at modulix]

Re: Too Many if Statements?

slogging_away wrote:[color=blue]
> bruno at modulix wrote:
>
>[color=green]
>>Looks like a memory problem then...[/color]
>
>
> The system I am using has 2GB of memory, (unless you are syaing the
> memory is faulty).[/color]

Nope, just that I don't know of any system with unlimited memory. BTW,
having 2GB of ram does not mean there are 2GB available for your program.

(And BTW, this is just one of the *possible* reasons of the problem
you've discovered. May be quite unrelated as well...)
[color=blue][color=green]
>>Why storing error messages ? Why don't you just write'em out (be it to
>>stdout or to a file) ?[/color]
>
> I guess I could do that, (write them to a file as they are discovered).[/color]

Well, this is how most programs do.
[color=blue]
> Right now the error messages are stored in the array and then the the
> array is scanned via, a for loop and the error messages are written to
> several files in different formats based on the severity of the errors,
> (on a per device basis, a per severity basis, etc.). This keeps the
> write statements to a minimum and in a central location of the script
> instead of having several statements for each individual error message
> spread throughout the script, (three write statements per error message
> at over 500 error messages would be a significant change).[/color]

This is a design problem, not an implementation problem. Just factor out
the part that do the error message dispatch into a function, then call
this function instead of storing the message. Or even better, use an
existing logging library...
[color=blue]
> Not to complain but if I can't use arrays then thats a pretty
> significant limitation.[/color]

Please understand that available memory is a limitation of the *system*
- you'd get the same limitation with *every* language ever (well, if you
know of a language that expand free ram ad infinitum, please let us know !-)


--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'onurb@xiludom. gro'.split('@')])"