Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!news.mathworks.com!europa.eng.gtefsd.com!howland.reston.ans.net!news.sprintlink.net!redstone.interpath.net!sas!mozart.unx.sas.com!saswss
From: saswss@hotellng.unx.sas.com (Warren Sarle)
Subject: Re: On computing # nodes in hidden layer...
Originator: saswss@hotellng.unx.sas.com
Sender: news@unx.sas.com (Noter of Newsworthy Events)
Message-ID: <D4FInv.5r1@unx.sas.com>
Date: Thu, 23 Feb 1995 01:26:19 GMT
X-Nntp-Posting-Host: hotellng.unx.sas.com
References:  <Mark_Vogt.15.001147CE@qmgate.anl.gov>
Organization: SAS Institute Inc.
Keywords: ANN Hidden layer
Lines: 55


In article <Mark_Vogt.15.001147CE@qmgate.anl.gov>, Mark_Vogt@qmgate.anl.gov (Mark Edward Vogt) writes:
|> Input layer nodes = M
|>
|> Hidden layer nodes = N := Sqrt (M*P)
|>
|> Output layer nodes = P

Usually Scott replies to this FAQ, but maybe he's getting sick of it,
so I'll take a turn.

All rules of thumb based on the numbers of inputs and outputs are utter,
unredeemable nonsense. A net with 1 input and 1 output may need
1,000,000 hidden nodes. A net with 1,000 inputs and 1,000 outputs may
need 1 (or zero) hidden nodes.  The optimal number of hidden nodes
depends on:

 * The number of training cases.

 * The amount of noise.

 * The desired accuracy of generalization.

 * The training method.

 * The type of activation functions.

 * Details of the function being learned, which are difficult to describe
   and which you probably don't know anyway.

 * Fill in the blank ____________________________________________

A discussion of practical methods for choosing the number of hidden
nodes would run dozens to hundreds of pages, but to summarize:

 * With stopped training or regularization, use _lots_ of hidden
   nodes.

 * Otherwise, start with 1 hidden node and work your way up, using
   some estimate of generalization error to make the decision.

Either way, it's also a good idea to consider direct connections from
inputs to outputs.







-- 

Warren S. Sarle       SAS Institute Inc.   The opinions expressed here
saswss@unx.sas.com    SAS Campus Drive     are mine and not necessarily
(919) 677-8000        Cary, NC 27513, USA  those of SAS Institute.
