Newsgroups: comp.lang.smalltalk
Path: cantaloupe.srv.cs.cmu.edu!rochester!udel!news.mathworks.com!usenet.eel.ufl.edu!interlog.com!news1.fonorola.net!fonorola!news!dbuck
From: dbuck@infoweb.magi.com (David Buck)
Subject: Re: Rounding Error in Digitalk 3.0.1
Sender: news@magi.com
Message-ID: <DAHvFI.2Kx@magi.com>
Date: Tue, 20 Jun 1995 23:18:54 GMT
References: <9505208036.AA803683043@smtp.bnr.com>
Nntp-Posting-Host: infoweb.magi.com
Organization: Magi Data Consulting
Lines: 34

In article <9505208036.AA803683043@smtp.bnr.com>,
Griff Lewis  <Griff_Lewis@SMTP.BNR.COM> wrote:
>     >I have had this experience in Digitalk 3.0.1
>
>     >0.025 printRounded:2 => 0.03
>     >1.025 printRounded:2 => 1.02
>
>     >Looks like a bug to me.
>     >Has anyone encountered this shared with me how to fix it ?
>
>     This also appears in Digitalk 2.1.2
>
>     It certainly is a bug. In fact it seems to round up for all numbers
>     except ones where the integer part of the float is either 1 or -1.

This is actually not a bug,  The basic problem here is that Floats are 
represented in binary to a finite precision. This means that in the 
internal representation, 0.025 becomes a repeating decimal in binary.  Of 
course, you can't represent a repeating decimal in standard floating 
point formats, so the system rounds up or down to the least significant 
bit. When this is printed back on the screen, the number will be slightly 
larger than 0.025 or slighly smaller.  Depending on which is the case, 
the rounding will round up or down.

If you want perfect accuracy, you need to use fractions or fixed point 
calculations.  Any language that uses floating point numbers represented 
in binary format will be inconsistent in this calculation.  The same 
problem occurs in C and BASIC.  (COBOL has BCD numbers that can retain 
accuracy).

David Buck
dbuck@magi.com
The Object People

