Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!oitnews.harvard.edu!purdue!lerc.nasa.gov!magnus.acs.ohio-state.edu!math.ohio-state.edu!howland.reston.ans.net!Germany.EU.net!ieunet!curia!csws5.ucc.ie!turlough
From: turlough@odyssey.ucc.ie (Turlough FitzPatrick)
Subject: Re: [Q] Purpose of extra hidden layers?
Message-ID: <DCq637.ArA@curia.ucc.ie>
Sender: usenet@curia.ucc.ie
Reply-To: turlough@odyssey.ucc.ie
Organization: Computer Science Dept., University College, Cork, Ireland
References: <gomes.14.3017B914@odie.ee.wits.ac.za>
Date: Thu, 3 Aug 1995 07:57:06 GMT
Lines: 41

In article 3017B914@odie.ee.wits.ac.za, gomes@odie.ee.wits.ac.za (Nando de Freitas) writes:
>Hi
>
>I often find that by using two hidden layers and a ridge construction 
>- y =sigma[sum(Wx+bias)] - I get better results (less parameters 
>required and faster convergence) than by using a single hidden layer. 
>Intuitively, one can sort of observe that by having a second hidden layer 
>one obtains more intricate combinations of the basis functions. Thus it 
>becomes easier to implement a particular nonlinear mapping.  
>
>Yet, I would like reasons, preferably expressed in a mathematical format, 
>that explain this behaviour. 
>
>Can anyone please recommend papers on the subject? Perhaps give your own 
>thoughts?
>
>Thanks
>Nando 

By the addition of a second hidden layer you are increasing the dimensionality
of the problem space. This means that the metwork has a larger area within
which to represent the relationship.  I dont know of any papers that give 
mathematical justification but it can make generalisation easier. The problem
with doing this is that you might give the network too much space, so much so
that the network can just learn off the data of the training set and not generalise at 
all. An easy way to ensure this has not happened in your case might be to compare
the performance of ethe network with two hidden layers to that of the network
with one.


---
===============================================================
_/_/_/_/_/_/_/_/_/_/_/=Turlough FitzPatrick,
        _/_/          =Computer Science Dept.,
      _/ _/           =University College Cork,
    _/_/_/_/_/_/      =Cork,       /   turlough@csvax1.ucc.ie 
  _/   _/             =Ireland.   /    turlough@cs.ucc.ie
_/    _/              =E-mail: --/     turlough@odyssey.ucc.ie
******* WWW at  http://odyssey.ucc.ie/www/turlough.html *******
===============================================================

