Results 1 to 4 of 4
  1. #1
    Join Date
    Jan 2016
    Posts
    7

    Lightbulb Answered: Formula "precision" results

    Hi

    I have some columns calculate with fomula . e.g. LN or COVARIANTE etc etc.. but I'm not able to have a results with a precision > 5.. I need to increase precision

    some test I did

    select LN(412.9 / 404.4) LNBYDIVISION, LN(1.021018793273986) LNFIXED, CAST(LN(1.021018793273986) AS DECIMAL(20, 9)) LNFIXEDCASTED, cast( LN(DECFLOAT(1.021018793273986, 34)) as decimal(20,9)) LNFIXEDDOCFLO from sysibm.sysdummy1

    but all columns are calculated as 0.0208

    but the more precise value is 0.020800945746

    do you have some idea?

    thanks to all

  2. Best Answer
    Posted by ARWinner

    "What DB2 version and OS are you using? How are you getting the results? If I take you statement and enter it into a CLP, I get the following:

    Code:
    select LN(412.9 / 404.4) LNBYDIVISION, LN(1.021018793273986) LNFIXED, CAST(LN(1.021018793273986) AS DECIMAL(20, 9)) LNFIXEDCASTED, cast( LN(DECFLOAT(1.021018793273986, 34)) as decimal(20,9)) LNFIXEDDOCFLO from sysibm.sysdummy1
    
    LNBYDIVISION             LNFIXED                  LNFIXEDCASTED          LNFIXEDDOCFLO
    ------------------------ ------------------------ ---------------------- ----------------------
      +2.08009457457281E-002   +2.08009457457281E-002            0.020800945            0.020800946
    
      1 record(s) selected.
    Andy"


  3. #2
    Join Date
    Jan 2003
    Posts
    4,292
    Provided Answers: 5
    What DB2 version and OS are you using? How are you getting the results? If I take you statement and enter it into a CLP, I get the following:

    Code:
    select LN(412.9 / 404.4) LNBYDIVISION, LN(1.021018793273986) LNFIXED, CAST(LN(1.021018793273986) AS DECIMAL(20, 9)) LNFIXEDCASTED, cast( LN(DECFLOAT(1.021018793273986, 34)) as decimal(20,9)) LNFIXEDDOCFLO from sysibm.sysdummy1
    
    LNBYDIVISION             LNFIXED                  LNFIXEDCASTED          LNFIXEDDOCFLO
    ------------------------ ------------------------ ---------------------- ----------------------
      +2.08009457457281E-002   +2.08009457457281E-002            0.020800945            0.020800946
    
      1 record(s) selected.
    Andy

  4. #3
    Join Date
    Jul 2013
    Location
    Moscow, Russia
    Posts
    666
    Provided Answers: 55
    Hi,

    Which db2 version do you have?

    Describe returns the following on my DB2:
    Code:
     SQL type              Type length  Column name                     Name length
     --------------------  -----------  ------------------------------  -----------
     480   DOUBLE                    8  LNBYDIVISION                             12
     480   DOUBLE                    8  LNFIXED                                   7
     484   DECIMAL               20, 9  LNFIXEDCASTED                            13
     484   DECIMAL               20, 9  LNFIXEDDOCFLO                            13
    The result is:
    Code:
    LNBYDIVISION             LNFIXED                  LNFIXEDCASTED          LNFIXEDDOCFLO         
    ------------------------ ------------------------ ---------------------- ----------------------
      +2,08009457457281E-002   +2,08009457457281E-002            0,020800945            0,020800946
    Regards,
    Mark.

  5. #4
    Join Date
    Jan 2016
    Posts
    7
    Hi All
    thanks and sorry

    First of all I'm using DB2 UDB 10.5.7 (workgroup)
    Really I'm using JDBC using a client SquirrelSQL.. and it seems this client have a problem because right now after your suggestion I run it on CLP and also from Data Studio (using JDBC) and I have the right precision

    Thanks to all

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •