2d array slicing problem

[jeg]

dear all,

i'm an astronomer working with 2d images -- 2d numarrays. i have a
script which basically does some operations on some images, and one of
the first steps is to find a galaxy on an image (at, say, a known x,y
coord), and create a sub-image by slicing out part of the larger array
to create a more managable smaller one (e.g. 50x50 pixels from a
1000x1000 array).

i was running this fine under python 2.3.4, but due to the slowness of
my machine, moved to a speedy 64-bit linux box running version 2.4.2.
arrays were no longer being sliced in the correct way, for example:

B = A[3:7,6:10]

generates different Bs if i run an identical script on the same image
on the different machines.

note: we deal with FITS format images, which are read into numarrays
using the pyfits.py module. does anyone know any obvious reason this
should be happening?

many thanks if you can help,

jim

[Robert Kern]

Re: 2d array slicing problem

jeg wrote:[color=blue]
> dear all,
>
> i'm an astronomer working with 2d images -- 2d numarrays. i have a
> script which basically does some operations on some images, and one of
> the first steps is to find a galaxy on an image (at, say, a known x,y
> coord), and create a sub-image by slicing out part of the larger array
> to create a more managable smaller one (e.g. 50x50 pixels from a
> 1000x1000 array).
>
> i was running this fine under python 2.3.4, but due to the slowness of
> my machine, moved to a speedy 64-bit linux box running version 2.4.2.
> arrays were no longer being sliced in the correct way, for example:
>
> B = A[3:7,6:10]
>
> generates different Bs if i run an identical script on the same image
> on the different machines.
>
> note: we deal with FITS format images, which are read into numarrays
> using the pyfits.py module. does anyone know any obvious reason this
> should be happening?[/color]

Not without having the data and the machines in front of me. You'll get
better luck asking on numarray's mailing list.

--
Robert Kern
rkern@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

[jepler@unpythonic.net]

Re: 2d array slicing problem

Do you have a simple program that demonstrates the problem?

I have an x86 machine with Python 2.3, and an x86_64 machine with Python 2.4
available. I wrote a simple test program which performs a slice operation,
but it behaves the same on both platforms.

Here's the program:
#------------------------------------------------------------------------
import numarray, sys, os
print "python: ", sys.version_inf o
print "numarray:" , numarray.__vers ion__
print "CPU: ", os.uname()[-1]

n = numarray.arange (10)
o = numarray.outerp roduct(n,n)
p = o[3:7,6:10].copy()
q = numarray.outerp roduct([3,4,5,6], [6,7,8,9])
print "Success: ", numarray.alltru e(p.ravel() == q.ravel())
#------------------------------------------------------------------------

Here are the two results I gathered:

python: (2, 4, 1, 'final', 0)
numarray: 1.3.3
CPU: x86_64
Success: True

python: (2, 3, 2, 'final', 0)
numarray: 0.9
CPU: i686
Success: 1

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDRsYYJd0 1MZaTXX0RAq35AJ 477QdItft2GAaL+ ElmqtbEE8fHQQCg pjpO
Vl4i9wURTugz61X fo5Puc1w=
=X7lq
-----END PGP SIGNATURE-----

[jeg]

Re: 2d array slicing problem

thanks, i ran it -- the only difference i got was the numarray version:
1.1.1 on the 686, and 1.3.3 on the 64bit... but i wouldn't have thought
that would make too much difference.

[Robert Kern]

Re: 2d array slicing problem

Tune Kamae (sent by Nabble.com) wrote:[color=blue]
> I am thinking to upgrade my desktop to 64bit cpu with 16GB memory to handle
> large astronomical images and data. I wonder if
> 1) the latest numarry (besides 2d slicing) has been tested with one or more
> 64 bit CPU and Linux distributions[/color]

Certainly.
[color=blue]
> 2) with 64 bit address space, will numarray be able to handle larger arrays
> and matrices (many 3d-arrays 100x100x100 and matrices 50k x 50k)?
> (with 32 bit CPU I was limited by the memory.)
>
> I would appreciate knowing about your experience.[/color]

You'll probably want to ask on the appropriate mailing list[1], but
since you ask here, I will say that there are limitations that prevent
numarray from fully utilizing 64-bit systems. numarray uses the Python
buffer interface which is addressed by 32-bit integers even on 64-bit
platforms[2]. There has been some work on numarray's replacement,
scipy_core, to address this deficiency, but more work needs to be done
and more volunteers with 64-bit systems are needed. 100x100x100 arrays
should work with numarray; 50000x50000 perhaps not. You could always try
and let us know.

[1] http://lists.sourceforge.net/lists/l...mpy-discussion
[2] http://permalink.gmane.org/gmane.com...c.general/2690

--
Robert Kern
robert.kern@gma il.com

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter