huge slowdown on append

[=?ISO-8859-1?B?aWFuYXLp?=]

Hey all,

I have a for loop which included the line:

items_ren.appen d(join(newPath, renamedItem))

I changed it to this:
items_ren.appen d([join(newPath,re namedItem), False])

And processing speed is now much much slower. For 5780 items the old
function would take 9.5 seconds (there is other stuff going on
obviously), after changing that single line, speed is now 55 seconds
in the same conditions!

Can anyone give me some pointers please? I would like to keep the same
structure if possible, as it allows the code in other places to be
much faster. TIA

BTW, the 'join' function is this:
def join(newPath,re namedItem):
return unicode(os.path .join(newPath,r enamedItem))

[=?ISO-8859-1?B?aWFuYXLp?=]

Re: huge slowdown on append

On Dec 21, 3:29 pm, "ianaré" <ian...@gmail.c omwrote:

Hey all,
>
I have a for loop which included the line:
>
items_ren.appen d(join(newPath, renamedItem))
>
I changed it to this:
items_ren.appen d([join(newPath,re namedItem), False])
>
And processing speed is now much much slower. For 5780 items the old
function would take 9.5 seconds (there is other stuff going on
obviously), after changing that single line, speed is now 55 seconds
in the same conditions!
>
Can anyone give me some pointers please? I would like to keep the same
structure if possible, as it allows the code in other places to be
much faster. TIA
>
BTW, the 'join' function is this:
def join(newPath,re namedItem):
return unicode(os.path .join(newPath,r enamedItem))

Seems like I found a way ... Use the old way, then at the end of the
for loop add the 'False'.
def addStatus(x):
return (x,False)
items_ren = map(addStatus, items_ren)

9.5 seconds wOOt!

However, if anyone has a faster way, let me know!

[Fredrik Lundh]

Re: huge slowdown on append

ianaré wrote:

>I changed it to this:
>items_ren.appe nd([join(newPath,re namedItem), False])
>>
>And processing speed is now much much slower. For 5780 items the old
>function would take 9.5 seconds (there is other stuff going on
>obviously), after changing that single line, speed is now 55 seconds
>in the same conditions!

have you checked the process size? short lists are relatively expensive
to create, both performance-wise, and more importantly, memory-wise.
try using tuples instead.

or better, move the flags to a separate data structure (either a list,
or, if possible, a set or dictionary keyed on filenames).

</F>

[Peter Otten]

Re: huge slowdown on append

ianaré wrote:

On Dec 21, 3:29 pm, "ianaré" <ian...@gmail.c omwrote:

>Hey all,
>>
>I have a for loop which included the line:
>>
>items_ren.appe nd(join(newPath ,renamedItem))
>>
>I changed it to this:
>items_ren.appe nd([join(newPath,re namedItem), False])
>>
>And processing speed is now much much slower. For 5780 items the old
>function would take 9.5 seconds (there is other stuff going on
>obviously), after changing that single line, speed is now 55 seconds
>in the same conditions!
>>
>Can anyone give me some pointers please? I would like to keep the same
>structure if possible, as it allows the code in other places to be
>much faster. TIA
>>
>BTW, the 'join' function is this:
>def join(newPath,re namedItem):
> return unicode(os.path .join(newPath,r enamedItem))

>
Seems like I found a way ... Use the old way, then at the end of the
for loop add the 'False'.
def addStatus(x):
return (x,False)
items_ren = map(addStatus, items_ren)
>
9.5 seconds wOOt!
>
However, if anyone has a faster way, let me know!

Maybe

import itertools as it
items_ren = zip(items_ren, it.repeat(False ))

Or if you want to start earlier on

def gen_items():
for ...
yield newPath, renamedItem

items_ren = zip(it.imap(uni code, it.starmap(os.p ath.join, gen_items())),
it.repeat(False ))

Probably buggy, but you should get the idea...

Peter