Newsgroups: comp.lang.smalltalk
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!nntp.sei.cmu.edu!news.psc.edu!hudson.lm.com!godot.cc.duq.edu!news.duke.edu!news.mathworks.com!newsfeed.internetmci.com!howland.reston.ans.net!ix.netcom.com!netcomsv!uu3news.netcom.com!netcomsv!uucp3.netcom.com!slcgate!servio!servio!aland
From: aland@servio.slc.com (Alan Darlington)
Subject: Re: VW 2.0 Number truncation
Message-ID: <1995Sep11.180429.21507@slc.com>
Sender: news@slc.com (USENET News)
Nntp-Posting-Host: servio
Organization: GemStone Systems, Inc., Beaverton OR, USA
References: <41mkkn$ns@natasha.rmii.com> <41qequ$11ku@watnews1.watson.ibm.com> <macgremd-050995095356@ck5121gm45b.open.ac.uk>
Date: Mon, 11 Sep 1995 18:04:29 GMT
Lines: 63

macgremd@bean.open.ac.uk (malcolm macgregor) writes:
> In article <41qequ$11ku@watnews1.watson.ibm.com>, David N. Smith
> <dnsmith@watson.ibm.com> wrote:
> 
> > In article <41mkkn$ns@natasha.rmii.com> Mike Fontenot, mfonteno@qadas.com
> > writes:
> > >I have recently begun working in VisualWorks 2.0 and have run into a
> > >problem converting strings to numbers.  The following code:
> > >
> > >'80000.65' asNumber   -->  80000.6
> > >
> > >truncates the cents portion of this value.  
> VW uses single precision floating point as the default. 
> > '8000.65' asNumber  "One less zero"
> works 
> But:
> 
> '80000.65' asNumber asDouble "gives 80000.6484375d"

There are two separate (and confusing) issues here:  (1) the internal
precision of floats and doubles for representing a number, and (2) the
way that these numbers are presented for inspection.

Issue (2) is causing the problems above.  (Note that if it were (1),
converting a string to a float would cause loss of precision which
would not at all be affected by subsequently converting it to a
double.  :-)  Look at implementors of #defaultNumberOfDigits.  The
default LimitedPrecisionReal method returns 6, while the Double method
returns 14!.  Check out the following results:


| n s |
s := WriteStream on: (String new: 20).
n := '80000.65' asNumber.
n printOn: s digits: 20.
^ s contents 
 '80000.64849853515625'

| n s |
s := WriteStream on: (String new: 20).
n := 80000.65.
n printOn: s digits: 20.
^ s contents 
 '80000.64849853515625'

| n s |
s := WriteStream on: (String new: 20).
n := 80000.65d.
n printOn: s digits: 20.
^ s contents 
 '80000.649999999993156d'


Once you get around the printing problem, the affects of the
internal precision of floats and doubles becomes much more
apparent!  :-)

Hope this reduces (rather than increases) the confusion.

  -- Alan
    (standard disclaimer)


