Results 1 to 3 of 3
  1. #1
    Join Date
    Mar 2009
    Posts
    1

    Unanswered: How cast to Float works

    There is a select query , which contains the formula that deals with currency value as,

    SELECT CAST((CAST(RecordCount as FLOAT(20) * CAST(Sum(Power(Amount,2)) as FLOAT(20)) - Power(Sum(Amount),2)) / CAST((RecordCount * (RecordCount- 1)) as FLOAT(20))

    Consider below are the values for this expression,

    SELECT CAST((57 * CAST(8056893945.093 as FLOAT(20)) - 207067699401.4860) / (57 * (57- 1)) as FLOAT(20))

    But in the above expression the CAST(8056893945.093 as FLOAT(20)) is giving the result as 8.056894E9 which is equivalent to 8056894000 right.

    Because of this truncation or rounding, the final value I am getting is 79002280 (If I calculate manually the exact result by zero decimal place is 79002273).

    Could any one please explain why the truncation or rounding off happens (My assumption here is value out of range) and how the rounding off is carried out in SQL Server in this type of case?

  2. #2
    Join Date
    Feb 2004
    Location
    One Flump in One Place
    Posts
    14,912
    You know that floats are floating point numbers right? They aren't really for storing absolute values.
    SQL Server Forums - Disasterous Decimal Debacle
    What Every Computer Scientist Should Know About Floating-Point Arithmetic
    Testimonial:
    pootle flump
    ur codings are working excelent.

  3. #3
    Join Date
    Feb 2004
    Location
    One Flump in One Place
    Posts
    14,912
    Testimonial:
    pootle flump
    ur codings are working excelent.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •