Problem With Precision

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Victor

    #1

    Problem With Precision

    When I loop through a datatable rows and add Amt field content (which
    is float in database) sometimes it adds upto High Precision value
    leaving a difference of -1.8189894035458 6E-12 though I am only adding
    Currency Format numbers i.e., 23.43,56.99 etc,So I am not able to
    directly compare the total(Computer value) to a value in text field.







    Here is the Code
    If Session("DT") Is Nothing Then
    Dim SQL As String = "SELECT AcctNum, Amount FROM
    Accts where 1=2"
    Dim Dbase As New DBInterface(DBC onnString)
    Session("DT") = Dbase.GetDataTa ble(SQL)
    End If
    Dim DT As DataTable
    DT = Session("DT")
    Dim Drow As DataRow = DT.NewRow
    Drow("AcctNum") = txtAcct.Text
    Drow("Amount") = txtAmt.Text
    DT.Rows.Add(Dro w)
    Dim Total As Double
    Dim EachRow As DataRow
    For Each EachRow In DT.Rows
    Total += EachRow("Amount ")
    Next

    dgrd.Columns.It em(0).FooterTex t = "Total"
    dgrd.Columns.It em(1).FooterTex t = Total '.ToString("c")
    txtTotal.Text = Total
    Response.Write( Total - CDbl(txtTotal.T ext))
    Session("DT") = DT
    dgrd.DataSource = DT
    dgrd.DataBind()

  • Chris Dunaway

    #2
    Re: Problem With Precision

    Instead of using Double, try using Decimal instead. Your problem
    appears because a floating point number cannot accurately be
    represented in binary. There have been numerous posts on these
    newsgroups which explains it.

    Comment

    • Victor

      #3
      Re: Problem With Precision

      Great looks like working...Thank s

      Comment

      Working...