Efficient custom checksum of one-dim numpy arrays of uint16

[Slaunger]

Hi,

This is my first post here, I am looking forward to being here.

I have actually posted almost the same question on comp.lang.pytho n:

http://groups.google.dk/group/comp.lang.python/browse_thread/thread/2f0e7ee3ad76d5a3/e2eae3719c6e3fe8?hl=en#e2eae3719c6e3fe8

and got some hints at what i could do, but it is a little bit too complicated for me (yet), and i was wondering if there is a simpler way to do this.

I have implemented two functions for making a complementary ones addition of uint16 one-dimensional arrays, because I need this for computing a checksum on very many arrays with 50-10000 elements (difrerent size for each array)

Here it goes

from numpy import *

def compl_add_uint1 6(a, b):
c = a + b
c += c >> 16
return c & 0xFFFF

def compl_one_check sum(uint16s):
return reduce(compl_ad d_uint16, uint16s, 0x0000)

This works correct, but it is also *the* bottleneck in the several ksloc apllication as 88% of the time is spend doing the reduce, and 95% of that time is spend in the compl_add_uint1 6 function.

I does not surprise me that it is slow as the interpreter has to walk through the three lines in compl_add_uint on each addition.

However, i cannot figure out how to make it the right pythonic and efficient way?

I prefer a pure numpy implementation, but I am also willing to go into some weave inlining and blitz, although I have never tried that before and I do not have a blitz compatible C++ compiler installed on my MS Server 2003 OS.

How can I make that faster?

From looking around it seems like i need to go in some ufunc reduce direction to do this the numpy way, but how?

Thank you for you time,

-- Slaunger

[Slaunger]

I got the answer on comp.python.num eric.general from Charles R. Harris.

This does the trick:

def complement_add_uint16(x) :
    y = sum(x, dtype=uint64)    
    while y >= 2**16 :
        y = (y & uint64(0xffff)) + (y >> uint64(16))
    return y

That just boosted my checksum processing speed from 413 kB/s to, hold on, 47400 kB/s, which is more than a 100-fold increase. numpy rules!

-- Slaunger