int() 24 times slower then long() in Python 2.3

[Willem]

When I run the follwing code using Python 2.3:

from time import clock
t1 = clock ()
for i in range (10000): a = int ('bbbbaaaa', 16)
t2 = clock ()
for i in range (10000): a = long ('bbbbaaaa', 16)
t3 = clock ()
print (t2-t1) / (t3-t2)

it prints out: 23.9673206147

That seems to mean that the int-conversion is 24 times slower than the
long conversion.
The reason is that in Python 2.3 the int-conversion generates
warning messages that you *never* see but that consume a *lot* of CPU.

So it may happen that old code performing well under Python 2.2 suddenly
slows down a considerable amount under Python 2.3 without any percievable
change in the output ...

Willem Vree

BTW.
The message that you don't see is: OverflowWarning : string/unicode conversion
and you can make it appear by starting the progam with[color=blue]
>Python -Wall script.py[/color]
or including the following lines in the source text:
import warnings
warnings.resetw arnings()

I suppose that Python is getting over-engineered by too professional
programmers. The first signs of decay?

[Peter Otten]

Re: int() 24 times slower then long() in Python 2.3

Willem wrote:
[color=blue]
> When I run the follwing code using Python 2.3:
>
> from time import clock
> t1 = clock ()
> for i in range (10000): a = int ('bbbbaaaa', 16)
> t2 = clock ()
> for i in range (10000): a = long ('bbbbaaaa', 16)
> t3 = clock ()
> print (t2-t1) / (t3-t2)
>
> it prints out: 23.9673206147
>
> That seems to mean that the int-conversion is 24 times slower than the
> long conversion.
> The reason is that in Python 2.3 the int-conversion generates
> warning messages that you *never* see but that consume a *lot* of CPU.[/color]
[color=blue]
> I suppose that Python is getting over-engineered by too professional
> programmers. The first signs of decay?[/color]

No. I think warning.py needs some _more_ engineering - throw in a dictionary
to speed up repetetive calls or so. In the meantime:

import warnings
def dummy(*args, **kw):
pass
warnings.warn = dummy

Now watch it speed up :)

Peter

[James Henderson]

Re: int() 24 times slower then long() in Python 2.3

On Tuesday 13 July 2004 3:09 pm, Willem wrote:[color=blue]
> When I run the follwing code using Python 2.3:
>
> from time import clock
> t1 = clock ()
> for i in range (10000): a = int ('bbbbaaaa', 16)
> t2 = clock ()
> for i in range (10000): a = long ('bbbbaaaa', 16)
> t3 = clock ()
> print (t2-t1) / (t3-t2)
>
> it prints out: 23.9673206147
>
> That seems to mean that the int-conversion is 24 times slower than the
> long conversion.
> The reason is that in Python 2.3 the int-conversion generates
> warning messages that you *never* see but that consume a *lot* of CPU.
>
> So it may happen that old code performing well under Python 2.2 suddenly
> slows down a considerable amount under Python 2.3 without any percievable
> change in the output ...[/color]

I don't think this code did work perfectly well under Python2.2 (see below),
or am I missing something?

Python 2.2.2 (#1, Feb 24 2003, 19:13:11)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-4)] on linux2
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> for i in range (10000): a = int ('bbbbaaaa', 16)[/color][/color][/color]
....
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: int() literal too large: bbbbaaaa[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

--
James Henderson, Logical Progression Ltd

http://www.logicalprogression.net

http://mailmanager.sourceforge.net

[Istvan Albert]

Re: int() 24 times slower then long() in Python 2.3

Willem wrote:

[color=blue]
> So it may happen that old code performing well under Python 2.2 suddenly
> slows down a considerable amount under Python 2.3 without any percievable
> change in the output ...[/color]

IMO the speed at which a bunch of invalid conversions are
executed means nothing at all. Could you come up with an example
that show the same symptoms in a meaningful context?

i.

[paolo veronelli]

Re: int() 24 times slower then long() in Python 2.3

On 13 Jul 2004 14:34:25 -0700, Paul Rubin <"http://phr.cx"@NOSPAM. invalid>
wrote:
[color=blue]
> "Nick Smallbone" <nick@nick8325. freeserve.co.uk > writes:[color=green][color=darkred]
>> >
>> > IMO the speed at which a bunch of invalid conversions are
>> > executed means nothing at all. Could you come up with an example
>> > that show the same symptoms in a meaningful context?[/color]
>>
>> What do you mean? bbbbaaaa is a hex number.
>>[color=darkred]
>> >>> int('bbbbaaaa', 16)[/color]
>> 3149638314L[/color]
>
> It's not an int. It has to attempt to convert to int, trap the error
> and recover from it, and then convert to a long.[/color]
I don't undersand the meaning of int()
If I want an int() (two bytes?) I want two bytes.It should truncate.If the
result is the same with long() a part from the warning
I think int() is unmeaningfull.T ransparency is away from warnings.
This python is saying I used the wrong function and there at least two
cases:
I have wrong results.
He is doing my businness.

IMHO I want wrong results.


--
.....lotta dura per la verdura

[James Henderson]

Re: int() 24 times slower then long() in Python 2.3

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wednesday 14 July 2004 8:32 am, paolo veronelli wrote:[color=blue]
> On 13 Jul 2004 14:34:25 -0700, Paul Rubin <"http://phr.cx"@NOSPAM. invalid>
>
> wrote:[color=green]
> > "Nick Smallbone" <nick@nick8325. freeserve.co.uk > writes:[color=darkred]
> >> > IMO the speed at which a bunch of invalid conversions are
> >> > executed means nothing at all. Could you come up with an example
> >> > that show the same symptoms in a meaningful context?
> >>
> >> What do you mean? bbbbaaaa is a hex number.
> >>
> >> >>> int('bbbbaaaa', 16)
> >>
> >> 3149638314L[/color]
> >
> > It's not an int. It has to attempt to convert to int, trap the error
> > and recover from it, and then convert to a long.[/color]
>
> I don't undersand the meaning of int()
> If I want an int() (two bytes?) I want two bytes.It should truncate.If the
> result is the same with long() a part from the warning
> I think int() is unmeaningfull.T ransparency is away from warnings.
> This python is saying I used the wrong function and there at least two
> cases:
> I have wrong results.
> He is doing my businness.
>
> IMHO I want wrong results.
>
>
> --
> ....lotta dura per la verdura[/color]

I think you should look at PEP 237 "Unifying Long Integers and Integers" - it
may even reassure you. As Peter Hansen has already hinted in reply to
another of your messages, Python is in the process of unifying longs and
ints.

It used to be that calling int() on a big number gave an exception - perhaps a
more Pythonic version of your requested giving the wrong result - and now
that it doesn't I suppose there is a sense in which the current distinction
between ints and longs is "unmeaningf ul" from the user's point of view (it's
not meaningless under the hood of course). According to the PEP the decision
to keep two types was that it:

is far easier to implement, is backwards
compatible at the C API level, and in addition can be implemented
partially as a transitional measure.

Perhaps you thing that ints and longs should not be unified but I won't start
arguing till you come out and say it. :)

James
- --
James Henderson, Logical Progression Ltd

http://www.logicalprogression.net

http://mailmanager.sourceforge.net

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFA9QYUd1y XQ13iWmgRAmmSAK DWOH+vv9RAvPape mqXDSTax+P1uQCg zbbc
AHRxEquLPpWE32t asFiJOsw=
=taJY
-----END PGP SIGNATURE-----

[paolo veronelli]

Re: int() 24 times slower then long() in Python 2.3

On Wed, 14 Jul 2004 11:08:16 +0100, James Henderson
<james@logicalp rogression.net> wrote:
[color=blue]
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Wednesday 14 July 2004 8:32 am, paolo veronelli wrote:[color=green]
>> On 13 Jul 2004 14:34:25 -0700, Paul Rubin
>> <"http://phr.cx"@NOSPAM. invalid>
>>
>> wrote:[color=darkred]
>> > "Nick Smallbone" <nick@nick8325. freeserve.co.uk > writes:
>> >> > IMO the speed at which a bunch of invalid conversions are
>> >> > executed means nothing at all. Could you come up with an example
>> >> > that show the same symptoms in a meaningful context?
>> >>
>> >> What do you mean? bbbbaaaa is a hex number.
>> >>
>> >> >>> int('bbbbaaaa', 16)
>> >>
>> >> 3149638314L
>> >
>> > It's not an int. It has to attempt to convert to int, trap the error
>> > and recover from it, and then convert to a long.[/color]
>>
>> I don't undersand the meaning of int()
>> If I want an int() (two bytes?) I want two bytes.It should truncate.If
>> the
>> result is the same with long() a part from the warning
>> I think int() is unmeaningfull.T ransparency is away from warnings.
>> This python is saying I used the wrong function and there at least two
>> cases:
>> I have wrong results.
>> He is doing my businness.
>>
>> IMHO I want wrong results.
>>[/color][/color]
[color=blue]
>
> I think you should look at PEP 237 "Unifying Long Integers and Integers"
> - it
> may even reassure you. As Peter Hansen has already hinted in reply to
> another of your messages, Python is in the process of unifying longs and
> ints.
>
> It used to be that calling int() on a big number gave an exception -
> perhaps a
> more Pythonic version of your requested giving the wrong result - and now
> that it doesn't I suppose there is a sense in which the current
> distinction
> between ints and longs is "unmeaningf ul" from the user's point of view
> (it's
> not meaningless under the hood of course). According to the PEP the
> decision
> to keep two types was that it:
>
> is far easier to implement, is backwards
> compatible at the C API level, and in addition can be implemented
> partially as a transitional measure.
>
> Perhaps you thing that ints and longs should not be unified but I won't
> start
> arguing till you come out and say it. :)
>
> James[/color]
I don't want to change any way the base of PEP 237,but this kind of
performance
flaw is a problem and it's possible to be the first of a long list.
It's good to know, always ,why we pay a price.Better an Error then a ghost
warning,which
is itself the cause of the flaw.
Python is fast ,this is why I use it.Anyone has a different goal in using
it.

Regards Paolino




--
.....lotta dura per la verdura

[James Henderson]

Re: int() 24 times slower then long() in Python 2.3

On Wednesday 14 July 2004 12:28 pm, paolo veronelli wrote:[color=blue]
>
> On Wed, 14 Jul 2004 11:08:16 +0100, James Henderson wrote:[color=green]
> >
> > I think you should look at PEP 237 "Unifying Long Integers and Integers"
> > - it
> > may even reassure you. As Peter Hansen has already hinted in reply to
> > another of your messages, Python is in the process of unifying longs and
> > ints.
> >
> > It used to be that calling int() on a big number gave an exception -
> > perhaps a
> > more Pythonic version of your requested giving the wrong result - and now
> > that it doesn't I suppose there is a sense in which the current
> > distinction
> > between ints and longs is "unmeaningf ul" from the user's point of view
> > (it's
> > not meaningless under the hood of course). According to the PEP the
> > decision
> > to keep two types was that it:
> >
> > is far easier to implement, is backwards
> > compatible at the C API level, and in addition can be implemented
> > partially as a transitional measure.
> >
> > Perhaps you thing that ints and longs should not be unified but I won't
> > start
> > arguing till you come out and say it. :)
> >
> > James[/color]
>
> I don't want to change any way the base of PEP 237,but this kind of
> performance
> flaw is a problem and it's possible to be the first of a long list.
> It's good to know, always ,why we pay a price.Better an Error then a ghost
> warning,which
> is itself the cause of the flaw.
> Python is fast ,this is why I use it.Anyone has a different goal in using
> it.
>
> Regards Paolino[/color]

I actually agree it's a shame that even ignored warnings have such a
performance impact but at least these warnings are scheduled to be removed
sometime in Python 2.4 (they're still there in 2.4a1 though). Apparently
they were added for "those concerned about extreme backwards compatibility".

James
--
James Henderson, Logical Progression Ltd

http://www.logicalprogression.net

http://mailmanager.sourceforge.net

[Willem]

Re: int() 24 times slower then long() in Python 2.3

James Henderson <james@logicalp rogression.net> wrote in message news:<mailman.3 73.1089807240.5 135.python-list@python.org >...[color=blue]
> On Wednesday 14 July 2004 12:28 pm, paolo veronelli wrote:[color=green]
> >
> > On Wed, 14 Jul 2004 11:08:16 +0100, James Henderson wrote:[color=darkred]
> > >
> > > I think you should look at PEP 237 "Unifying Long Integers and Integers"
> > > - it
> > > may even reassure you. As Peter Hansen has already hinted in reply to
> > > another of your messages, Python is in the process of unifying longs and
> > > ints.
> > >
> > > It used to be that calling int() on a big number gave an exception -
> > > perhaps a
> > > more Pythonic version of your requested giving the wrong result - and now
> > > that it doesn't I suppose there is a sense in which the current
> > > distinction
> > > between ints and longs is "unmeaningf ul" from the user's point of view
> > > (it's
> > > not meaningless under the hood of course). According to the PEP the
> > > decision
> > > to keep two types was that it:
> > >
> > > is far easier to implement, is backwards
> > > compatible at the C API level, and in addition can be implemented
> > > partially as a transitional measure.
> > >
> > > Perhaps you thing that ints and longs should not be unified but I won't
> > > start
> > > arguing till you come out and say it. :)
> > >
> > > James[/color]
> >
> > I don't want to change any way the base of PEP 237,but this kind of
> > performance
> > flaw is a problem and it's possible to be the first of a long list.
> > It's good to know, always ,why we pay a price.Better an Error then a ghost
> > warning,which
> > is itself the cause of the flaw.
> > Python is fast ,this is why I use it.Anyone has a different goal in using
> > it.
> >
> > Regards Paolino[/color]
>
> I actually agree it's a shame that even ignored warnings have such a
> performance impact but at least these warnings are scheduled to be removed
> sometime in Python 2.4 (they're still there in 2.4a1 though). Apparently
> they were added for "those concerned about extreme backwards compatibility".
>
> James[/color]

That was indeed the reason of my original posting: As a casual
programmer I was disturbed that upgrading to Python 2.3 actually
slowed down my program significantly and that I had to spend so much
time to find the reason: somebody has decided to hide these particular
overflow warnings ...
Why warning and then suppressing it by default?
I understand the problem will disappear in the next version.
I apologize for my remark on "over-engineering". I posted the message
when I was still angry, having spent so much time in finding the "bug"

Willem

[James Henderson]

Re: int() 24 times slower then long() in Python 2.3

On Thursday 15 July 2004 9:58 am, Willem wrote:[color=blue]
> That was indeed the reason of my original posting: As a casual
> programmer I was disturbed that upgrading to Python 2.3 actually
> slowed down my program significantly[/color]

As I said before, I don't think the particular code you posted would have
worked at all before 2.3.
[color=blue]
> and that I had to spend so much
> time to find the reason: somebody has decided to hide these particular
> overflow warnings ...
> Why warning and then suppressing it by default?[/color]

The warnings are there because some people may want to know about when the
type conversion is happening, since it's a new feature.

They're suppressed by default because most people won't want to know. :)

J.
--
James Henderson, Logical Progression Ltd

http://www.logicalprogression.net

http://mailmanager.sourceforge.net

[Willem]

Re: int() 24 times slower then long() in Python 2.3

James Henderson <james@logicalp rogression.net> wrote in message news:<mailman.4 26.1089886380.5 135.python-list@python.org >...
[color=blue]
> As I said before, I don't think the particular code you posted would have
> worked at all before 2.3.[/color]

No, indeed. I tried to make a small program that demonstrates the
problem.
But since you are interested, I have made another more accurate
demonstration of my problem.

Willem

--------------------demonstration---------------------------
# we want to read lots of 32 bit CRC values form a text file
# the numbers are hexadecimal ASCII-strings and have to be
# converted to 32-bit signed integers (which happens to be
# the output of zlib.crc32)
import warnings
from time import clock

# be compatible with Python 2.2 for integer overflow
warnings.filter warnings ('error', category=Overfl owWarning)

# my original 8char-hex to 32bit-int conversion in Python 2.2
# it runs much faster for positive numbers than for negative ones
# but is on average faster then an if-based version
a = -158933416 # an example number, make it positive to see quite
# different results
b = '%08x' % a # the hex ascii string
print a, b
t1 = clock()
for i in range (10000):
try: c = int (b, 16)
except: c = - int (0x100000000 - long (b, 16))
t2 = clock()
print t2 - t1, c, type (c)

# straight forward adaptation to Python 2.3: replace try by if.
# first we set warnings to the default
# (this version does not work in Python 2.2)
warnings.filter warnings ('ignore', category=Overfl owWarning)
t1 = clock()
for i in range (10000):
c = int (b, 16)
if c > 0x7fffffff: c = - int (0x100000000 - c)
t2 = clock()
print t2 - t1, c, type (c)

# this adaptation runs 10 times faster then the previous version
# in Python 2.3, but is on average slower than the first version
# in Python 2.2. This is certainly not an obvious adaptation.
# Nobody would expect a long conversion to be, on average, faster
# than an int conversion for 32 bit numbers ...
t1 = clock()
for i in range (10000):
c = long (b, 16)
if c > 0x7fffffff: c = - int (0x100000000 - c)
else: c = int(c)
t2 = clock()
print t2 - t1, c, type (c)