preallocate list

[Jim]

Re: preallocate list

F. Petitjean wrote:[color=blue]
> Le Wed, 13 Apr 2005 16:46:53 +0100, Jim a écrit :
>[color=green]
>>What I really want is a Numeric array but I don't think Numeric supports
>>importing files.[/color]
>
> Numeric arrays can be serialized from/to files through pickles :
> import Numeric as N
> help(N.load)
> help(N.dump)
> (and it is space efficient)
>[color=green]
>>Jim[/color][/color]
Yeah thanks. I'm generating them using Matlab though so I'd have to get
the format the same. I use Matlab because I get the results I want. When
I get to know Python + scipy etc. better I might remove that step.

Thanks again

Jim

[Jim]

Re: preallocate list

Steven Bethard wrote:[color=blue]
> Jim wrote:
>[color=green]
>> What I really want is a Numeric array but I don't think Numeric
>> supports importing files.[/color]
>
>
> Hmmm... Maybe the scipy package?
>
> I think scipy.io.read_a rray might help, but I've never used it.
>
> STeVe[/color]
Sounds promising.

I only got Numeric because I wanted scipy but I've hardly explored it as
I kept running into problems even with the complicated examples cut and
paste into a file ;)

Oh yeah, I wanted to explore the GA module but no docs :( and I got busy
doing other stuff.

Thanks

Jim

[Jim]

Re: preallocate list

[color=blue]
> ivec = n*[None]
>
> so that if I use a list element before intializing it, for example
>
> ivec[0] += 1
>
> I get an error message
>
> File "xxnone.py" , line 2, in ?
> ivec[0] += 1
> TypeError: unsupported operand type(s) for +=: 'NoneType' and 'int'
>
> This is in the same spirit as Python's (welcome) termination of a
> program when one tries to use an uninitalized scalar variable.
>[/color]

I feel foolish that I forgot about *. I've just started with Python then
took 2 weeks off. I'll explore pre-allocation when I'm back up to speed.

Yep, I use None a lot.

Thanks

Jim

[Dan Christensen]

Re: preallocate list

Bill Mill <bill.mill@gmai l.com> writes:
[color=blue]
> Bill Mill <bill.mill@gmai l.com> writes:
>[color=green]
>> I would profile your app to see that it's your append which is taking
>> ages, but to preallocate a list of strings would look like:
>>
>> ["This is an average length string" for i in range(approx_le ngth)][/color][/color]

I don't think there's any point putting strings into the preallocated
list. A list is just an array of pointers to objects, so any object
will do fine for preallocation, no matter what the list will be used for.
[color=blue][color=green]
>> My guess is that it won't help to preallocate, but time it and let us
>> know. A test to back my guess:
>>
>> import timeit, math
>>
>> def test1():
>> lst = [0 for i in range(100000)]
>> for i in xrange(100000):
>> lst[i] = math.sin(i) * i
>>
>> def test2():
>> lst = []
>> for i in xrange(100000):
>> lst.append(math .sin(i) * i)[/color][/color]

....
[color=blue]
> The results change slightly when I actually insert an integer, instead
> of a float, with lst[i] = i and lst.append(i):
>
> 09:14 AM ~$ python test.py
> time1: 3.352000
> time2: 3.672000[/color]

If you use

lst = range(100000)

or even better

lst = [None]*100000

then test1 is more than twice as fast as test2:

time1: 2.437730
time2: 5.308054

(using python 2.4).

Your code

lst = [0 for i in range(100000)]

made python do an extra 100000-iteration loop.

Dan

[John Machin]

Re: preallocate list

On Wed, 13 Apr 2005 14:28:51 +0100, Jim <jbo@cannedham. ee.ed.ac.uk>
wrote:
[color=blue]
>Thanks for the suggestions. I guess I must ensure that this is my bottle
>neck.
><code>
> def readFactorsInto List(self,filen ame,numberLoads ):[/color]

1. "numberLoad s" is not used.
[color=blue]
> factors = []
> f = open(self.based ir + filename,'r')
> line = f.readline()
> tokens = line.split()
> columns = len(tokens)
> if int(columns) == number:[/color]

2. "columns" is already an int (unless of course you've redefined
"len"!). Doing int(columns) is pointless.
3. What is "number"? Same as "numberLoad s"?
4. Please explain in general what is the layout of your file and in
particular, what is the significance of the first line of the file and
of the above "if" test.
[color=blue]
> for line in f:
> factor = []
> tokens = line.split()
> for i in tokens:
> factor.append(f loat(i))[/color]

4. "factor" is built and then not used any more??
[color=blue]
> factors.append( loadFactor)[/color]

5. What is "loadFactor "? Same as "factor"?
[color=blue]
> else:
> for line in f:
> tokens = line.split()
> factors.append([float(tokens[0])] * number)[/color]

6. You throw away any tokens in the line after the first??
[color=blue]
> return factors
></code>
>
>OK. I've just tried with 4 lines and the code works.[/color]

Which code works? The code you posted? Please define "works".

[color=blue]
> With 11000 lines it
>uses all CPU for at least 30 secs. There must be a better way.[/color]

Perhaps after you post the code that you've actually run, and
explained what your file layout is, and what you are trying to
achieve, then we can give you some meaningful help.

Cheers,

John

[Jim]

Re: preallocate list

John Machin wrote:[color=blue]
> On Wed, 13 Apr 2005 14:28:51 +0100, Jim <jbo@cannedham. ee.ed.ac.uk>
> wrote:
>
>[color=green]
>>Thanks for the suggestions. I guess I must ensure that this is my bottle
>>neck.
>><code>
>> def readFactorsInto List(self,filen ame,numberLoads ):[/color]
>
>
> 1. "numberLoad s" is not used.
>
>[color=green]
>> factors = []
>> f = open(self.based ir + filename,'r')
>> line = f.readline()
>> tokens = line.split()
>> columns = len(tokens)
>> if int(columns) == number:[/color]
>
>
> 2. "columns" is already an int (unless of course you've redefined
> "len"!). Doing int(columns) is pointless.
> 3. What is "number"? Same as "numberLoad s"?
> 4. Please explain in general what is the layout of your file and in
> particular, what is the significance of the first line of the file and
> of the above "if" test.
>
>[color=green]
>> for line in f:
>> factor = []
>> tokens = line.split()
>> for i in tokens:
>> factor.append(f loat(i))[/color]
>
>
> 4. "factor" is built and then not used any more??
>
>[color=green]
>> factors.append( loadFactor)[/color]
>
>
> 5. What is "loadFactor "? Same as "factor"?
>
>[color=green]
>> else:
>> for line in f:
>> tokens = line.split()
>> factors.append([float(tokens[0])] * number)[/color]
>
>
> 6. You throw away any tokens in the line after the first??
>
>[color=green]
>> return factors
>></code>
>>
>>OK. I've just tried with 4 lines and the code works.[/color]
>
>
> Which code works? The code you posted? Please define "works".
>
>
>[color=green]
>>With 11000 lines it
>>uses all CPU for at least 30 secs. There must be a better way.[/color]
>
>
> Perhaps after you post the code that you've actually run, and
> explained what your file layout is, and what you are trying to
> achieve, then we can give you some meaningful help.
>
> Cheers,
>
> John
>
>
>[/color]

Thanks for looking John. For that I should take a little time to explain.

I tried to rename the variables, some of them were four words long. I
got a couple of the renames wrong. Sorry.

Regarding 'works'. I meant that with a text file of four lines the code
completed. With my desired size 11000 lines it didn't complete within
the limits of my patience. I didn't try any other size.

Also I perhaps wrongly use the newsgroup threads paradigm in trying to
restart my query with extra information (that turned out a little faulty).

Luckily the other branches yielded fruit.

Thanks again
Jim