need optimizing help

[rabbits77]

I have a dictionary with a very very large(possibly millions) of
key/value pairs.
The key is a tuple that looks like (id,date)
What is the fastest way to get out all of the values that match any
key given that they individual key elements are coming from two
seperate lists?
The approach of
for id in IDS:
for date in dates:
data=myDict[(id,date)]

seems to just take too long. Is there a speedier, more pythonic, way
of doing this? Any help speeding this up would be much appreciated!!

[Riccardo Galli]

Re: need optimizing help

On Sat, 13 Mar 2004 10:07:54 -0800, rabbits77 wrote:
[color=blue]
> I have a dictionary with a very very large(possibly millions) of
> key/value pairs.
> The key is a tuple that looks like (id,date)
> What is the fastest way to get out all of the values that match any
> key given that they individual key elements are coming from two
> seperate lists?
> The approach of
> for id in IDS:
> for date in dates:
> data=myDict[(id,date)]
>
> seems to just take too long. Is there a speedier, more pythonic, way
> of doing this? Any help speeding this up would be much appreciated!![/color]

if IDS and dates lenght is the same, you could write
for i in enumerate(IDS):
data=myDict[(IDS[i],dates[i])]

which is more good looking, but I think performances are similar.

my 2 cents.

Riccardo

[Aahz]

Re: need optimizing help

In article <68aadb4a.04031 31007.8eeeb26@p osting.google.c om>,
rabbits77 <rabbits77@bigm ailbox.net> wrote:[color=blue]
>
>I have a dictionary with a very very large(possibly millions) of
>key/value pairs.[/color]

If it's literally millions, you probably want to use a database. Just
the dictionary itself (not even talking about the objects used for keys
and values) will take at least 12MB per million entries on a machine with
32-bit addresses.
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

"usenet imitates usenet" --Darkhawk

[William Park]

Re: need optimizing help

rabbits77 <rabbits77@bigm ailbox.net> wrote:[color=blue]
> I have a dictionary with a very very large(possibly millions) of
> key/value pairs.
> The key is a tuple that looks like (id,date)
> What is the fastest way to get out all of the values that match any
> key given that they individual key elements are coming from two
> seperate lists?
> The approach of
> for id in IDS:
> for date in dates:
> data=myDict[(id,date)]
>
> seems to just take too long. Is there a speedier, more pythonic, way
> of doing this? Any help speeding this up would be much appreciated!![/color]

Hmm... since you're looking up every possible combination of 'id' and
'date', there is no short cut of algorithm. As for speed, it may be
faster to use external database.

--
William Park, Open Geometry Consulting, <opengeometry@y ahoo.ca>
Linux solution for data processing and document management.