If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

 
Go Back  dBforums > Database Server Software > Sybase > Comparing the results of float data type to numeric data type Math operations.

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1 (permalink)  
Old 03-19-03, 10:13
JoeLynch JoeLynch is offline
Registered User
 
Join Date: Mar 2003
Location: Woking, UK
Posts: 1
Angry Comparing the results of float data type to numeric data type Math operations.

The following happens on Sybase 11.9.2 and on 12.5 to a lesser extent.

We have a mathematical function, as shown below, that aims to check that @pt is an exact multiple of @ts:

-- select
-- @fl_s = @ts,
-- @fl_t = @pt

-- select
-- @fl_total = @fl_t / @fl_s,
-- @int_total = @pt / @ts

-- if @fl_total != @int_total
-- begin
-- select @error_message =
-- "@pt must be a multiple of @ts"
-- return 1
-- end

Sybase gets a little confused because floats are an approximate data type and errors are generated if, for example, @pt is set as 0.001 and @ts is set as 0.000001 but is then stored as 0.000009999999999, on the server. When the operation is done, with these values, Sybase complains that @pt isn't an exact multiple of @ts (i.e. 0.001 is not an exact multiple of 0.000009999999999).

Short of changing to a numeric data type for both values (this is being looked into but would involved an extremely large amount of work), does anyone have any ideas on how the operation could be done to eliminate the error?

I have looked into converting the values from floats, into numerics and then doing the calculation but this doesn't always work.

I have also looked into using the "round()" arithmetic operation and this works well until you start using larger values of @pt (arithmetic overflows and other errors occur).

Any ideas?

Cheers,

Joe.
Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On