Warnings killing my performance

[Kylotan]

I have the following code:

def IntToRandFloat( x):
"""Given a 32-bit integer, return a float in [-1.0, 1.0]"""
x = int(x)
x = int(x << 13) ^ x
return (1.0-((x*(x*x*15731+ 789221)+1376312 589)&0x7fffffff )/1073741824.0)


Basically it's a function directly copied from a C implementation. Now
it appears that the line with the left-shift causes the "losing bits
or changing sign will return a long in Python 2.4 and up" warning. All
well and good - I explicitly cast it back to an int so that the code
won't break when I upgrade Python. I also disable the warning using
warnings.filter warnings(action = 'ignore', etc).

However when the time comes to profile my app, I see lines like these:

11266617 function calls in 488.717 CPU seconds
ncalls tottime percall cumtime percall
filename:lineno (function)
3145728 129.744 0.000 364.845 0.000
terraingen.py:2 2(IntToRandFloa t)
3142872 150.484 0.000 235.101 0.000 warnings.py:24( warn)
3142872 84.617 0.000 84.617 0.000
warnings.py:59( warn_explicit)

Now I obviously can't afford to have almost half my program time
consumed by warnings that I don't want to see.

It gets stranger. If I change the shift line to this:
x = int(x * (2 ** 13)) ^ x
x = int(x | 0xFFFF)

....it's still calling 'warn' once for each time IntToRandFloat is
called, but I see no warning appear, even when I don't import warnings
and disable any. I have no other imports (that i can see) which might
be disabling a warning behind the scenes.

So I have these problems: warnings are slow (even when disabled),
sometimes warnings are being issued and I never see them, and given
that I never see the warnings I don't know how to get around them.

So, my first question is to ask if there is a more efficient way of
disabling warnings?

And my second is, is there a quick way of taking an integer, shifting
it left 13 bits (or multiplying by 2 ** 13, whatever), discarding any
excess bits, and which won't cause a warning?

Lastly, a suggestion; if 2.4 will introduce an automatic promotion to
a long as a result of this shift operation, will the standard library
provide a C implementation of the lossy shift operator for those of us
that would benefit from a quick version? I'm guessing that promoting
it to a long and then getting it back to a normal int is not exactly
the speediest operation.

--
Ben Sizer

[Skip Montanaro]

Re: Warnings killing my performance


Ben> So I have these problems: warnings are slow (even when disabled),
Ben> sometimes warnings are being issued and I never see them, and given
Ben> that I never see the warnings I don't know how to get around them.

Ben> So, my first question is to ask if there is a more efficient way of
Ben> disabling warnings?

This may seem silly, but switching to the CVS version of Python (aka 2.4a0)
should help immensely, simply because that is gone:

% python2.3 ~/local/bin/timeit.py -s 'import sys ; sys.path.append ("/Users/skip/tmp"); from warn import IntToRandFloat' 'IntToRandFloat (789221)'
/Users/skip/tmp/warn.py:4: FutureWarning: x<<y losing bits or changing sign will return a long in Python 2.4 and up
x = int(x << 13) ^ x
10000 loops, best of 3: 114 usec per loop
% python2.4 ~/local/bin/timeit.py -s 'import sys ; sys.path.append ("/Users/skip/tmp"); from warn import IntToRandFloat' 'IntToRandFloat (789221)'
100000 loops, best of 3: 16.7 usec per loop

Ben> Lastly, a suggestion; if 2.4 will introduce an automatic promotion
Ben> to a long as a result of this shift operation, will the standard
Ben> library provide a C implementation of the lossy shift operator for
Ben> those of us that would benefit from a quick version? I'm guessing
Ben> that promoting it to a long and then getting it back to a normal
Ben> int is not exactly the speediest operation.

Apparently it's quite a bit faster than navigating all the warning
machinery. ;-)

Skip

[Kylotan]

Re: Warnings killing my performance

The 'or' in "x = int(x | 0xFFFF)" was obviously meant to be an 'and',
before anybody chooses to point it out. :)

--
Kylotan

[Terry Reedy]

Re: Warnings killing my performance


"Kylotan" <kylotan@hotmai l.com> wrote in message
news:153fa67.04 02060933.13840e 8c@posting.goog le.com...[color=blue]
> And my second is, is there a quick way of taking an integer, shifting
> it left 13 bits (or multiplying by 2 ** 13, whatever), discarding any
> excess bits, and which won't cause a warning?[/color]

Have you tried masking off the upper 13 bits *before* the shift? So that
there is no overflow to warn about? Does this make any sense?

TJR

[Kylotan]

Re: Warnings killing my performance

"Terry Reedy" <tjreedy@udel.e du> wrote in message news:<mailman.1 322.1076118385. 12720.python-list@python.org >...[color=blue]
> "Kylotan" <kylotan@hotmai l.com> wrote in message
> news:153fa67.04 02060933.13840e 8c@posting.goog le.com...[color=green]
> > And my second is, is there a quick way of taking an integer, shifting
> > it left 13 bits (or multiplying by 2 ** 13, whatever), discarding any
> > excess bits, and which won't cause a warning?[/color]
>
> Have you tried masking off the upper 13 bits *before* the shift? So that
> there is no overflow to warn about? Does this make any sense?[/color]

Yes, it makes perfect sense... but unfortunately getting rid of that
specific warning isn't enough. As noted in my first post, something is
going through the warnings mechanism silently. I changed the code to
this:

def IntToRandFloat( x):
"""Given a 32-bit integer, return a float in [-1.0, 1.0]"""
x = int(x) & 0x3FFFF # Preserve lowest 18 bits, to remove
overflow.
x = int(x << 13) ^ x # Shift left 13 bits.
return (1.0-((x*(x*x*15731+ 789221)+1376312 589)&0x7fffffff )/1073741824.0)

And the profiler is still showing a call to warn() and warn_explicit()
for each call to IntToRandFloat. I'm also having to preserve 18 bits
instead of the desired 19 because the warning displays when I use 18,
presumably because it includes the sign bit. However now I think I am
damaging the distribution of the function (which is supposed to
generate deterministic noise).

I probably should have mentioned earlier that I'm using "Python 2.3.3
(#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32" for
what it's worth.

If I knew what warning it was, maybe I could turn it into an error to
find out what's causing it. But right now, it's a silent warning that
I can't do much about. And which is slowing down my code a lot!

I'm starting to think that I might need to write a 1-function C++
extension here, which would be a shame.

--
Ben Sizer

[Peter Otten]

Re: Warnings killing my performance

Kylotan wrote:
[color=blue]
> I have the following code:
>
> def IntToRandFloat( x):
> """Given a 32-bit integer, return a float in [-1.0, 1.0]"""
> x = int(x)
> x = int(x << 13) ^ x
> return
> (1.0-((x*(x*x*15731+ 789221)+1376312 589)&0x7fffffff )/1073741824.0)
>
>
> Basically it's a function directly copied from a C implementation. Now
> it appears that the line with the left-shift causes the "losing bits
> or changing sign will return a long in Python 2.4 and up" warning. All
> well and good - I explicitly cast it back to an int so that the code
> won't break when I upgrade Python. I also disable the warning using
> warnings.filter warnings(action = 'ignore', etc).
>
> However when the time comes to profile my app, I see lines like these:
>
> 11266617 function calls in 488.717 CPU seconds
> ncalls tottime percall cumtime percall
> filename:lineno (function)
> 3145728 129.744 0.000 364.845 0.000
> terraingen.py:2 2(IntToRandFloa t)
> 3142872 150.484 0.000 235.101 0.000 warnings.py:24( warn)
> 3142872 84.617 0.000 84.617 0.000
> warnings.py:59( warn_explicit)
>
> Now I obviously can't afford to have almost half my program time
> consumed by warnings that I don't want to see.
>
> It gets stranger. If I change the shift line to this:
> x = int(x * (2 ** 13)) ^ x
> x = int(x | 0xFFFF)
>
> ...it's still calling 'warn' once for each time IntToRandFloat is
> called, but I see no warning appear, even when I don't import warnings
> and disable any. I have no other imports (that i can see) which might
> be disabling a warning behind the scenes.
>
> So I have these problems: warnings are slow (even when disabled),
> sometimes warnings are being issued and I never see them, and given
> that I never see the warnings I don't know how to get around them.
>
> So, my first question is to ask if there is a more efficient way of
> disabling warnings?[/color]

Have you tried brute force?

....> timeit.py -s"import profwarn" "profwarn.IntTo RandFloat(99)"
10000 loops, best of 3: 27.3 usec per loop

....> timeit.py -s"import profwarn;profwa rn.disableWarni ngs()"
"profwarn.IntTo RandFloat(99)"
100000 loops, best of 3: 7.8 usec per loop

with IntToRandFloat( ) copied* from your post and the following to entirely
disable warnings:

def disableWarnings ():
def _theevilunwarne r(*args):
pass
import warnings
warnings.warn = _theevilunwarne r
warnings.warn_e xplicit = _theevilunwarne r

I think the last line isn't necessary, but it won't do any (additional)
harm.

Peter

(*) Your email client seems to replace normal space with evil lookalikes, so
I had to delete and reinsert the entire whitespace.

[Andrew Bennetts]

Re: Warnings killing my performance

On Sat, Feb 07, 2004 at 04:45:12AM -0800, Kylotan wrote:[color=blue]
> specific warning isn't enough. As noted in my first post, something is
> going through the warnings mechanism silently. I changed the code to[/color]

Have you tried "python -Wall"? It shows PendingDeprecat ionWarnings, which
are usually not displayed.

-Andrew.

[Mel Wilson]

Re: Warnings killing my performance

In article <153fa67.040207 0445.77121b2e@p osting.google.c om>,
kylotan@hotmail .com (Kylotan) wrote:[color=blue]
>"Terry Reedy" <tjreedy@udel.e du> wrote in message news:<mailman.1 322.1076118385. 12720.python-list@python.org >...[color=green]
>> "Kylotan" <kylotan@hotmai l.com> wrote in message
>> news:153fa67.04 02060933.13840e 8c@posting.goog le.com...[color=darkred]
>> > And my second is, is there a quick way of taking an integer, shifting
>> > it left 13 bits (or multiplying by 2 ** 13, whatever), discarding any
>> > excess bits, and which won't cause a warning?[/color]
>>
>> Have you tried masking off the upper 13 bits *before* the shift? So that
>> there is no overflow to warn about? Does this make any sense?[/color]
>
>Yes, it makes perfect sense... but unfortunately getting rid of that
>specific warning isn't enough. As noted in my first post, something is
>going through the warnings mechanism silently. I changed the code to
>this:
>
>def IntToRandFloat( x):
> """Given a 32-bit integer, return a float in [-1.0, 1.0]"""
> x = int(x) & 0x3FFFF # Preserve lowest 18 bits, to remove
>overflow.
> x = int(x << 13) ^ x # Shift left 13 bits.
> return (1.0-((x*(x*x*15731+ 789221)+1376312 589)&0x7fffffff )/1073741824.0)
>
>And the profiler is still showing a call to warn() and warn_explicit()
>for each call to IntToRandFloat. I'm also having to preserve 18 bits
>instead of the desired 19 because the warning displays when I use 18,
>presumably because it includes the sign bit. However now I think I am
>damaging the distribution of the function (which is supposed to
>generate deterministic noise).[/color]

I notice that you're effectively cubing x, so you could
only keep 10 bits of it and be sure of avoiding 32-bit
overflow. Maybe a linear pseudo-random formula would save
some trouble. But it appears that random.randint also
generates invisible warnings.

The warnings go away with
x = long(x << 13) ^ x


Regards. Mel.

[Kylotan]

Re: Warnings killing my performance

Peter Otten <__peter__@web. de> wrote in message news:<c02ubl$qq t$07$1@news.t-online.com>...
[color=blue]
> def disableWarnings ():
> def _theevilunwarne r(*args):
> pass
> import warnings
> warnings.warn = _theevilunwarne r
> warnings.warn_e xplicit = _theevilunwarne r[/color]

Heh, that's great. However, I don't think it really scales well to a
full application, does it? :)
[color=blue]
> (*) Your email client seems to replace normal space with evil lookalikes, so
> I had to delete and reinsert the entire whitespace.[/color]

I'm posting through Google Groups; my ISP doesn't carry more than the
last 6 threads of comp.lang.pytho n, or indeed many other newsgroups. I
have no idea why.

--
Ben Sizer

[Kylotan]

Re: Warnings killing my performance

mwilson@the-wire.com (Mel Wilson) wrote in message news:<cFQJAls/KbZC089yn@the-wire.com>...
[color=blue]
> I notice that you're effectively cubing x, so you could
> only keep 10 bits of it and be sure of avoiding 32-bit
> overflow.[/color]

I'm not sure why you say that. If I feed '10' into the formula in
question I'll get something much larger than 1000 out!
[color=blue]
> Maybe a linear pseudo-random formula would save
> some trouble. But it appears that random.randint also
> generates invisible warnings.[/color]

Well, I changed the whole function to the following:

rnd.seed(x)
return rnd.uniform(-1.0, 1.0)

Where rnd is an instance of Random(). This is a little quicker, and
doesn't throw any warnings (visible or otherwise). I just have to hope
the random.uniform implementation doesn't change any time soon.

I would also hope that someone will find a way to remove that hidden
warning from randint because it seems to be an unnecessary performance
hit on what is otherwise a fairly fundamental function.
[color=blue]
> The warnings go away with
> x = long(x << 13) ^ x[/color]

So do the results I'm trying to get ;)

--
Ben Sizer

[Peter Otten]

Re: Warnings killing my performance

Kylotan wrote:
[color=blue]
> Peter Otten <__peter__@web. de> wrote in message
> news:<c02ubl$qq t$07$1@news.t-online.com>...
>[color=green]
>> def disableWarnings ():
>> def _theevilunwarne r(*args):
>> pass
>> import warnings
>> warnings.warn = _theevilunwarne r
>> warnings.warn_e xplicit = _theevilunwarne r[/color]
>
> Heh, that's great. However, I don't think it really scales well to a
> full application, does it? :)[/color]

Well, you could disable warnings in critical parts only, but that wouldn't
play well with threads. Enter the not so evil unwarner:

def disableWarnings ():
originalWarn = warnings.warn
disabled = sets.Set([
id(exceptions.O verflowWarning) ,
id(exceptions.F utureWarning)])

def _thenotsoevilun warner(message, category=None, stacklevel=1):
if id(category) in disabled:
#print "DISMISS", message, category
pass
else:
#print "PROPAGATE" , message, category
originalWarn(me ssage, category, stacklevel)
warnings.warn = _thenotsoevilun warner

It's still pretty fast (10.8 vs 27.1 usec on my machine), but the duplicate
test will slow down warnings that are actually propagated.

And if you are really ambitious, you could devise a patch to speed up warn()
in the library, maybe using a similar technique as shown above.

Peter