Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!howland.reston.ans.net!gatech!concert!sas!mozart.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: Range for Input-/outputdata?
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <CzC2Ir.36t@unx.sas.com>
Date: Tue, 15 Nov 1994 23:56:51 GMT
References:  <39q58t$4v1@fileserv.aber.ac.uk>
Nntp-Posting-Host: hotellng.unx.sas.com
Organization: SAS Institute Inc.
Lines: 43


In article <39q58t$4v1@fileserv.aber.ac.uk>, dbk@aber.ac.uk (Douglas B. Kell) writes:
|> In article <Cyyn48.LsL@unx.sas.com> saswss@hotellng.unx.sas.com (Warren Sarle) writes:
|> >...
|> >Standardization tends to make the training process better behaved by
|> >improving the numerical condition of the optimization problem and
|> >ensuring that certain default values involved in initialization and
|> >termination are appropriate. For linear models fitted by least-squares,
|> >the results are invariant or equivariant under addition or
|> >multiplication of the variables by a constant; hence there is no
|> >disadvantage to standardization. Invariance or equivariance holds
|> >similarly for standardization of the inputs in a feedforward NN.
|>
|> [chomp]
|> There is an implicit rider in here that may not be obvious (we
|> went into this a bit) unless made explicit. If this normalisation
|> (of inouts first) is between 0 and 1, the question arises as to
|> whether one normalises over the whole dataset of inputs (all columns
|> in a matrix of inputs representing training and test sets, and cross-
|> validation if a separate one is used) or whether EACH column is
|> normalised individually. This becomes significant where the range
|> in different columns is very different and where a single column with
|> big values would have the effect of squishing the variance of the
|> rest of columns. Warren appears to be talking about normalization over
|> a whole dataset, i.e. all columns together.

Actually I was referring to standardizing each column separately,
which is what "standardizing variables" means in statistics.
Standardizing over the whole matrix would accomplish very little.

|> ...
|> is that scaling each column over its range can have a dramatic effect
|> on learning in standard BP, effectively by increasing the variance of
|> columns whose absolute magnitude may be small.

Or decreasing the variance of columns whose absolute magnitude may be
large.

-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
