Hypergeometric distribution

[Raven]

Re: Hypergeometric distribution


Bengt Richter wrote:
[color=blue]
> ISTM you wouldn't get zero if you scaled by 10**significant _digits (however many
> you require) before dividing. E.g., expected hits per trillion (or septillion or whatever)
> expresses probability too. Perhaps that could work in your calculation?
>
> Regards,
> Bengt Richter[/color]

Sorry Bengt but I can't figure out how to do it, can you give me an
example please? Thanks in advance.
Ale

[Raven]

Re: Hypergeometric distribution


Bengt Richter wrote:

[color=blue]
> ISTM you wouldn't get zero if you scaled by 10**significant _digits (however many
> you require) before dividing. E.g., expected hits per trillion (or septillion or whatever)
> expresses probability too. Perhaps that could work in your calculation?
>
> Regards,
> Bengt Richter[/color]

Sorry but I can't figure out how to do it, can you give me an example
please?
Thnx

Ale

[Travis E. Oliphant]

Re: Hypergeometric distribution

Raven wrote:[color=blue]
> Thanks Steven for your very interesting post.
>
> This was a critical instance from my problem:
>
>[color=green][color=darkred]
>>>>from scipy import comb
>>>
>>>>comb(14354, 174)[/color][/color]
>
> inf
>
> The scipy.stats.dis tributions.hype rgeom function uses the scipy.comb
> function, so it returned nan since it tries to divide an infinite. I
> did not tried to write a self-made function using standard python as I
> supposed that the scipy functions reached python's limits but I was
> wrong, what a fool :-)[/color]

Notice the keyword for the comb function (in scipy) lets you use it to
compute exact values. SciPy does not just automatically use the long
integer because this will always slow you down.

comb(N, k, exact=0)

Combinations of N things taken k at a time.

If exact==0, then floating point precision is used, otherwise
exact long integer is computed.

Notes:
- Array arguments accepted only for exact=0 case.
- If k > N, N < 0, or k < 0, then a 0 is returned.

-Travis Oliphant

[Raven]

Re: Hypergeometric distribution


Travis E. Oliphant wrote:
[color=blue]
> Notice the keyword for the comb function (in scipy) lets you use it to
> compute exact values. SciPy does not just automatically use the long
> integer because this will always slow you down.
>
> comb(N, k, exact=0)
>
> Combinations of N things taken k at a time.
>
> If exact==0, then floating point precision is used, otherwise
> exact long integer is computed.
>
> Notes:
> - Array arguments accepted only for exact=0 case.
> - If k > N, N < 0, or k < 0, then a 0 is returned.
>
> -Travis Oliphant[/color]

Great, thanks Travis.
Ale

[Cameron Laird]

Re: Hypergeometric distribution

In article <1136201733.518 543.95060@f14g2 000cwb.googlegr oups.com>,
Raven <balckraven@gma il.com> wrote:[color=blue]
>Well, what to say? I am very happy for all the solutions you guys have
>posted :-)
>For Paul:
>I would prefer not to use Stirling's approximation
>
>
>The problem with long integers is that to calculate the hypergeometric
>I need to do float division and multiplication because integer division
>returns 0. A solution could be to calculate log(Long_Factor ial_Integer)[/color]

[Raven]

Re: Hypergeometric distribution

Cameron Laird wrote:
[color=blue]
> This thread confuses me.
>
> I've lost track of the real goal. If it's an exact calculation of
> binomial coefficients--or even one of several other potential
> targets mentioned--I echo Steven D'Aprano, and ask, are you *sure*
> the suggestions already offered aren't adequate?[/color]

Hi Cameron, my real goal was to calculate the hypergeometric
distribution. The problem was that the function for hypergeometric
calculation from scipy uses the scipy.comb function which by default
uses floats so for large numbers comb(n,r) returns inf. and hence the
hypergeometric returns nan.
The first suggestion, the one by Robert Kern, resolved my problem:

Raven wrote:[color=blue]
>Thanks to all of you guys, I could resolve my problem using the
>logarithms as proposed by Robert.[/color]

Then the other guys gave alternative solutions so I tried them out. So
form me the suggestions offered are more than adequate :-)

Cameron Laird wrote:[color=blue]
>Also, I think you
> might not realize how accurate Stirling's approximation (perhaps to
> second order) is in the range of interest.[/color]

The problem with Stirling's approximation is that I need to calculate
the hypergeometric hence the factorial for numbers within a large range
e.g. choose(14000,17 0) or choose(5,2)

Ale

[Paul Rubin]

Re: Hypergeometric distribution

"Raven" <balckraven@gma il.com> writes:[color=blue]
> The problem with Stirling's approximation is that I need to calculate
> the hypergeometric hence the factorial for numbers within a large range
> e.g. choose(14000,17 0) or choose(5,2)[/color]

Stirling's approximation to second order is fairly accurate even at
low values:

from math import log,exp,pi

def stirling(n):
# approx log(n!)
return n*(log(n)-1) + .5*(log(2.*pi*n )) + 1/(12.*n)

[color=blue][color=green][color=darkred]
>>> for n in range(1,6): print n, exp(stirling(n) )[/color][/color][/color]
...
1 1.00227444918
2 2.00065204769
3 6.00059914247
4 24.0010238913
5 120.002637086[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

To third order it's even better:

from math import log,exp,pi

def stirling(n):
# approx log(n!)
return n*(log(n)-1) + .5*(log(2.*pi*n )) + 1/(12.*n) - 1/(360.*n*n*n)

[color=blue][color=green][color=darkred]
>>> for n in range(1,6): print n, exp(stirling(n) )[/color][/color][/color]
...
1 0.999494216712
2 1.99995749743
3 5.99998182863
4 23.9999822028
5 119.999970391[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

Reference: http://en.wikipedia.or g/wiki/Stirling's_appr oximation

[Bengt Richter]

Re: Hypergeometric distribution

On 4 Jan 2006 12:46:47 -0800, "Raven" <balckraven@gma il.com> wrote:
[color=blue]
>Cameron Laird wrote:
>[color=green]
>> This thread confuses me.
>>
>> I've lost track of the real goal. If it's an exact calculation of
>> binomial coefficients--or even one of several other potential
>> targets mentioned--I echo Steven D'Aprano, and ask, are you *sure*
>> the suggestions already offered aren't adequate?[/color]
>
>Hi Cameron, my real goal was to calculate the hypergeometric
>distribution . The problem was that the function for hypergeometric[/color]
ISTM that can't have been your "real goal" -- unless you are e.g. preparing numeric
tables for publication. IOW, IWT you probably intend to USE the hypergeometric
distribution values in some useful way to go towards your "real goal." ;-)

The requirements of this USE are still not apparent to me in your posts, though
that may be because I've missed something.
[color=blue]
>calculation from scipy uses the scipy.comb function which by default
>uses floats so for large numbers comb(n,r) returns inf. and hence the
>hypergeometr ic returns nan.
>The first suggestion, the one by Robert Kern, resolved my problem:
>
>Raven wrote:[color=green]
>>Thanks to all of you guys, I could resolve my problem using the
>>logarithms as proposed by Robert.[/color]
>
>Then the other guys gave alternative solutions so I tried them out. So
>form me the suggestions offered are more than adequate :-)
>
>Cameron Laird wrote:[color=green]
>>Also, I think you
>> might not realize how accurate Stirling's approximation (perhaps to
>> second order) is in the range of interest.[/color]
>
>The problem with Stirling's approximation is that I need to calculate
>the hypergeometric hence the factorial for numbers within a large range
>e.g. choose(14000,17 0) or choose(5,2)
>[/color]
It seems you are hinting at some accuracy requirements that you haven't
yet explained. I'm curious how you use the values, and how that affects your
judgement of Stirling's approximation. In fact, perhaps the semantics of your
value usage could even suggest an alternate algorithmic approach to your actual end result.

Regards,
Bengt Richter

[Robert Kern]

Re: Hypergeometric distribution

Bengt Richter wrote:[color=blue]
> On 4 Jan 2006 12:46:47 -0800, "Raven" <balckraven@gma il.com> wrote:[/color]
[color=blue][color=green]
>>The problem with Stirling's approximation is that I need to calculate
>>the hypergeometric hence the factorial for numbers within a large range
>>e.g. choose(14000,17 0) or choose(5,2)[/color]
>
> It seems you are hinting at some accuracy requirements that you haven't
> yet explained. I'm curious how you use the values, and how that affects your
> judgement of Stirling's approximation. In fact, perhaps the semantics of your
> value usage could even suggest an alternate algorithmic approach to your actual end result.[/color]

Does it matter? Implementing Stirling's approximation is pointless when
scipy.special.g ammaln() or scipy.special.g amma() does it for him.

--
Robert Kern
robert.kern@gma il.com

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

[Bengt Richter]

Re: Hypergeometric distribution

On Thu, 05 Jan 2006 09:47:02 -0600, Robert Kern <robert.kern@gm ail.com> wrote:
[color=blue]
>Bengt Richter wrote:[color=green]
>> On 4 Jan 2006 12:46:47 -0800, "Raven" <balckraven@gma il.com> wrote:[/color]
>[color=green][color=darkred]
>>>The problem with Stirling's approximation is that I need to calculate
>>>the hypergeometric hence the factorial for numbers within a large range
>>>e.g. choose(14000,17 0) or choose(5,2)[/color]
>>
>> It seems you are hinting at some accuracy requirements that you haven't
>> yet explained. I'm curious how you use the values, and how that affects your
>> judgement of Stirling's approximation. In fact, perhaps the semantics of your
>> value usage could even suggest an alternate algorithmic approach to your actual end result.[/color]
>
>Does it matter? Implementing Stirling's approximation is pointless when
>scipy.special. gammaln() or scipy.special.g amma() does it for him.
>[/color]
Who's talking about implementing Stirling's approximation? ;-) I'm trying to determine first
why the OP is thinking there's a problem with using it at all. With "alternate algorithmic
approach" I didn't mean an alternate way of calculating Stirling's approximation. I meant
to allude to the possibility that pulling a little further on the requirements thread might
even unravel some of the rationale for calculating the hypergeometric per se, depending on
how he's actually using it and why. Same old, same old: requirements, requirements ;-)

Regards,
Bengt Richter