Numeric data type issue

[JP]

Why in the world is this happening:

double MyCount=0.00;
MyCount = (100 * 2611508) / 37792454;
Response.Write( MyCount.ToStrin g("F2");

Always return a whole # (in this case 6). The value should be:
6.9101307896015 432075408492922 952
(as large as the Windows calc would display).

I really want the 6.91 or rounded to the nearest 10th
Ive tried defining MyCount as float, double, decimal. Any string format I
use either "N2", "F2", "P", always returns .00

What am I doing wrong? grrrrrrr






--
JP
..NET Software Develper

[Oliver Sturm]

Re: Numeric data type issue

JP wrote:
[color=blue]
>Why in the world is this happening:
>
>double MyCount=0.00;
>MyCount = (100 * 2611508) / 37792454;
>Response.Write (MyCount.ToStri ng("F2");
>
>Always return a whole # (in this case 6). The value should be:
>6.910130789601 543207540849292 2952
>(as large as the Windows calc would display).
>
>I really want the 6.91 or rounded to the nearest 10th
>Ive tried defining MyCount as float, double, decimal. Any string format I
>use either "N2", "F2", "P", always returns .00[/color]

The compiler thinks that the numbers you are using in your calculation are
integer values, so the calculation is performed with integers, regardless
of the "target type". If you want the calculation to use floating point
values, you should use the "f" modifier with at least one, or better all
of the values. Like this:

MyCount = (100f * 2611508) / 37792454;

or (better because it doesn't depend on internal calculation order)

MyCount = (100f * 2611508f) / 37792454f;


Oliver Sturm
--
Expert programming and consulting services available
See http://www.sturmnet.org (try /blog as well)

[JP]

Re: Numeric data type issue

So if:

int GroupCount=2611 508;
int TotalCount = 37792454;
double MyCount=0.00;

then how do I use the f modifer with a veriable name
Actual caculation looks like this:
MyCount=(100f *GroupCount)/TotalCount

thx


"Oliver Sturm" wrote:
[color=blue]
> JP wrote:
>[color=green]
> >Why in the world is this happening:
> >
> >double MyCount=0.00;
> >MyCount = (100 * 2611508) / 37792454;
> >Response.Write (MyCount.ToStri ng("F2");
> >
> >Always return a whole # (in this case 6). The value should be:
> >6.910130789601 543207540849292 2952
> >(as large as the Windows calc would display).
> >
> >I really want the 6.91 or rounded to the nearest 10th
> >Ive tried defining MyCount as float, double, decimal. Any string format I
> >use either "N2", "F2", "P", always returns .00[/color]
>
> The compiler thinks that the numbers you are using in your calculation are
> integer values, so the calculation is performed with integers, regardless
> of the "target type". If you want the calculation to use floating point
> values, you should use the "f" modifier with at least one, or better all
> of the values. Like this:
>
> MyCount = (100f * 2611508) / 37792454;
>
> or (better because it doesn't depend on internal calculation order)
>
> MyCount = (100f * 2611508f) / 37792454f;
>
>
> Oliver Sturm
> --
> Expert programming and consulting services available
> See http://www.sturmnet.org (try /blog as well)
>[/color]

[Oliver Sturm]

Re: Numeric data type issue

JP wrote:
[color=blue]
>int GroupCount=2611 508;
>int TotalCount = 37792454;
>double MyCount=0.00;
>
>then how do I use the f modifer with a veriable name
>Actual caculation looks like this:
>MyCount=(100 f *GroupCount)/TotalCount[/color]

Well, in this specific case it should work fine, I think. But other than
that, if you actually had no numeric values left in your calculation and
only integer variables to deal with, you'd need to insert one or more
casts to make sure the right types are used for the calculation. Like here:

int GroupCount=2611 508;
int TotalCount = 37792454;
int hundred = 100;
double MyCount=0.00;

MyCount = ((double) hundred * GroupCount) / TotalCount;


Oliver Sturm
--
Expert programming and consulting services available
See http://www.sturmnet.org (try /blog as well)

[JP]

Re: Numeric data type issue

Dun nevermind I had a brain fart there:
[color=blue]
> int GroupCount=0f;
> int TotalCount = 0f;[/color]
double MyCount=0.00f

thx for your help :);


--
JP
..NET Software Develper


"JP" wrote:
[color=blue]
> So if:
>
> int GroupCount=2611 508;
> int TotalCount = 37792454;
> double MyCount=0.00;
>
> then how do I use the f modifer with a veriable name
> Actual caculation looks like this:
> MyCount=(100f *GroupCount)/TotalCount
>
> thx
>
>
> "Oliver Sturm" wrote:
>[color=green]
> > JP wrote:
> >[color=darkred]
> > >Why in the world is this happening:
> > >
> > >double MyCount=0.00;
> > >MyCount = (100 * 2611508) / 37792454;
> > >Response.Write (MyCount.ToStri ng("F2");
> > >
> > >Always return a whole # (in this case 6). The value should be:
> > >6.910130789601 543207540849292 2952
> > >(as large as the Windows calc would display).
> > >
> > >I really want the 6.91 or rounded to the nearest 10th
> > >Ive tried defining MyCount as float, double, decimal. Any string format I
> > >use either "N2", "F2", "P", always returns .00[/color]
> >
> > The compiler thinks that the numbers you are using in your calculation are
> > integer values, so the calculation is performed with integers, regardless
> > of the "target type". If you want the calculation to use floating point
> > values, you should use the "f" modifier with at least one, or better all
> > of the values. Like this:
> >
> > MyCount = (100f * 2611508) / 37792454;
> >
> > or (better because it doesn't depend on internal calculation order)
> >
> > MyCount = (100f * 2611508f) / 37792454f;
> >
> >
> > Oliver Sturm
> > --
> > Expert programming and consulting services available
> > See http://www.sturmnet.org (try /blog as well)
> >[/color][/color]

[JP]

Re: Numeric data type issue

that too lol
must be Monday again LOL

--
JP
..NET Software Develper


"Oliver Sturm" wrote:
[color=blue]
> JP wrote:
>[color=green]
> >int GroupCount=2611 508;
> >int TotalCount = 37792454;
> >double MyCount=0.00;
> >
> >then how do I use the f modifer with a veriable name
> >Actual caculation looks like this:
> >MyCount=(100 f *GroupCount)/TotalCount[/color]
>
> Well, in this specific case it should work fine, I think. But other than
> that, if you actually had no numeric values left in your calculation and
> only integer variables to deal with, you'd need to insert one or more
> casts to make sure the right types are used for the calculation. Like here:
>
> int GroupCount=2611 508;
> int TotalCount = 37792454;
> int hundred = 100;
> double MyCount=0.00;
>
> MyCount = ((double) hundred * GroupCount) / TotalCount;
>
>
> Oliver Sturm
> --
> Expert programming and consulting services available
> See http://www.sturmnet.org (try /blog as well)
>[/color]

[Peter van der Goes]

Re: Numeric data type issue


"JP" <JP@discussions .microsoft.com> wrote in message
news:24CFDE1E-0008-4468-908F-134FB64B3CF7@mi crosoft.com...[color=blue]
> Why in the world is this happening:
>
> double MyCount=0.00;
> MyCount = (100 * 2611508) / 37792454;
> Response.Write( MyCount.ToStrin g("F2");
>
> Always return a whole # (in this case 6). The value should be:
> 6.9101307896015 432075408492922 952
> (as large as the Windows calc would display).
>
> I really want the 6.91 or rounded to the nearest 10th
> Ive tried defining MyCount as float, double, decimal. Any string format I
> use either "N2", "F2", "P", always returns .00
>
> What am I doing wrong? grrrrrrr
>[/color]
All the numeric literals in your calculation are integers. Division with
integers yields an integer result *before* the result is stored. Storing the
result in a double cannot restore the lost precision.
Try this instead:

MyCount = (100.0 * 2611508.0) / 37792454.0;

Now you are working with double precision floating point literals.

Peter [MVP Visual Developer]
Jack of all trades, master of none.