Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!newsfeed.pitt.edu!gatech!newsfeed.internetmci.com!newsxfer2.itd.umich.edu!agate!sunsite.doc.ic.ac.uk!lboro.ac.uk!usenet
From: Dave the Troll <eldb3@lboro.ac.uk>
Subject: Re: [Q]Weight Decay: Why weights should be small?
Sender: usenet@lboro.ac.uk (Usenet-News)
Message-ID: <31CE70B2.B7B@lboro.ac.uk>
Date: Mon, 24 Jun 1996 10:40:50 GMT
X-Nntp-Posting-Host: elpc4726.lut.ac.uk
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=us-ascii
References: <4qiudl$g62@sand.cis.ufl.edu>
Mime-Version: 1.0
X-Mailer: Mozilla 2.01 (Win95; I)
Organization: Loughborough University, UK.
Lines: 18

Hyeoncheol Hon Kim wrote:
> 
> Why the magnitude of weights should be kept small?
> 
> Whats wrong with large-valued weights?

It is generally considered that a smooth solution is more likely 
to generalise to the required function rather than overfit to the 
data.  High weights imply high curve gradients and so unsmooth 
functions.  This in turn leads to possible overfitting of the 
network.

Dave Barnett
   _____
  /     \   Optical Engineering Group
__\_|\|_/_________________________________
    |/|
            Loughborough University
