Division Error?

[rhl]

I just came across this weird division result with VB.NET.

2/3 = 0.6666666666666 6663

With VB6, I get

2/3 =0.666666666666 667

Using the Windows Calculator results in
0.6666666666666 666666666666666 6667

I'm sure I'm not the first one to notice this...what gives?

rhl

[Scott M.]

Re: Division Error?

The differences are due to the math being done on different data types.

VB 6's, VB .NET's and Windows don't all have comprable data types.


"rhl" <rahulblr@gmail .comwrote in message
news:1185313429 .785058.17350@z 24g2000prh.goog legroups.com...

>I just came across this weird division result with VB.NET.
>
2/3 = 0.6666666666666 6663
>
With VB6, I get
>
2/3 =0.666666666666 667
>
Using the Windows Calculator results in
0.6666666666666 666666666666666 6667
>
I'm sure I'm not the first one to notice this...what gives?
>
rhl
>

[Patrice]

Re: Division Error?

Exact code ? I've got the same result than in VB6 with Debug.WriteLine (2 /
3)

Anyway, all those results are wrong and obviously you'll never be able to
display or even compute the correct result. More generally this is a well
know problem with "real" numbers.

You have several ways to workaround this depending on the problem you have
with this. You could use a better type (such as decimal) for increasing (and
possibly "guaranteed " precision), you could store this a a fraction etc...

--
Patrice


"rhl" <rahulblr@gmail .coma écrit dans le message de news:
1185313429.7850 58.17350@z24g20 00...legro ups.com...

>I just came across this weird division result with VB.NET.
>
2/3 = 0.6666666666666 6663
>
With VB6, I get
>
2/3 =0.666666666666 667
>
Using the Windows Calculator results in
0.6666666666666 666666666666666 6667
>
I'm sure I'm not the first one to notice this...what gives?
>
rhl
>