Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!newsfeed.pitt.edu!gatech!newsfeed.internetmci.com!howland.reston.ans.net!math.ohio-state.edu!jussieu.fr!pasteur.fr!univ-lyon1.fr!in2p3.fr!swidir.switch.ch!swsbe6.switch.ch!news.unige.ch!usenet
From: squire@cui.unige.ch (David SQUIRE)
Subject: Bias nodes: explanation
Message-ID: <1996Jan18.082808.28116@news.unige.ch>
Sender: usenet@news.unige.ch
Reply-To: squire@cui.unige.ch
Organization: University of Geneva, Switzerland
References: <4dkene$go2@alpha.sky.net>
Date: Thu, 18 Jan 1996 08:28:08 GMT
Lines: 39

In article <4dkene$go2@alpha.sky.net>, Sunil Jain <sunil@solar.sky.net> writes:
>Can somebody please outline in one or more sentences, the function of bias node(s) 
>in an artificial neural-network ?
>

The bias node is a dummy node that always has an output of 1. All nodes have it as
one of their inputs. The reason for this is so that the decision surfaces that
partition the input space do not have to pass through the origin. Without a bias
term, the activation of a node j with N input nodes is (in LaTeX format):

net_j = \sum_{i=1}^{N}{w_{ij}y_i}

where the y_i are the outputs of the input nodes. Note that if all the y_i = 0, then
the activation net_j = 0. No amount of training (i.e. modifying the w_{ij}) can
change this. Clearly this is an unwanted, and potentially disastrous, limitation on
the functions that the network can learn (think of the XOR problem).

This can be rectified by adding a constant term to the equation, which we will call
w_{0j}, so that:

net_j = \sum_{i=1}^{N}{w_{ij}y_i} + w_{0j}

Now the activation function does not have to pass through the origin. If we imagine
that there is a "bias node", which always has an output of 1, and we call that node
y_0, then we can write the above equation for the activation in the compact form:

net_j = \sum_{i=0}^{N}{w_{ij}y_i}

If you like, you can think of the bias weight as being a "threshold for firing" for
the neuron, as, if negative, it determines a value that the weighted inputs must
exceed before the activation becomes positive.

I hope that is clear.

Squizz




