Re: float numbers
very interesting..
I don't know...
very interesting..
I don't know...
-
-
Re: float numbers
(homa@havet.se) writes:[color=blue]
> I can't use the convert since I don't know how many decimals to round
> the result to and for this function that I'm writing the result must be
> exact.[/color]
In such case you should not use float. Float gives you approxamite numbers.
Float values consists of a 53-bit number with a mantissa. This permits
for a broad range of value, to the price of approxamite precision.
The given example:
declare @a float
declare @b int
set @a=0.9616458
set @b=60
print @a*@b
select @a*@b
Gives you 57.6987 for the print, because there is an implicit conversion
to string in SQL Server.
The SELECT statement returns a binary float value to the client, so it
up to the client how it is presented. In Query Analyzer I get
57.698748000000 002, whereas in Management Studio that ships with
SQL 2005, I get 57,698748. ISQL, the command-line tool that uses
DB-Library to connect returns 57.698748.
[color=blue]
> This is a GPS conversion formula and if I round things always
> with fir examole 6 decimals then the x,y coordinats will be wrong for
> the GPS system. In my example above I need the result to be exactley
> 57,698748 and not 57.6987 or 57.698747999999 114 .... Since I don't
> know the amount of decimlas returned I cant use the convert..[/color]
If you don't know the decimals of your result before hand, you will
have to convert the result to a string, and then try to guess how many
decimals you really have.
--
Erland Sommarskog, SQL Server MVP, esquel@sommarsk og.se
Books Online for SQL Server 2005 at
Books Online for SQL Server 2000 at
Comment
Comment