Newsgroups: comp.ai.neural-nets
From: David@longley.demon.co.uk (David Longley)
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!howland.reston.ans.net!news.sprintlink.net!pipex!peernews.demon.co.uk!longley.demon.co.uk!David
Subject: Re: linear separable boolean functions -- lists?
References: <3makuv$jng@agate.berkeley.edu>
Organization: None
Reply-To: David@longley.demon.co.uk
X-Newsreader: Newswin Alpha 0.7
Lines:  62
X-Posting-Host: longley.demon.co.uk
Date: Sun, 16 Apr 1995 15:34:41 +0000
Message-ID: <685278761wnr@longley.demon.co.uk>
Sender: usenet@demon.co.uk

Here is an extract from something I wrote recently. It refers to a 
text by Wasserman where he cites an old paper. I do not have the book 
at hand but will post the reference when I do. Be warned, I think it 
may have only been a conference presentation in the 1960s

From a pattern recognition or classification stance, it is  known 
that as the number of predicates increase, the number of linearly 
separable  functions becomes proportionately smaller as  is  made 
clear  by  the  following  extract  from  Wasserman  (1989)  when 
discussing the concept of linear separability:

    'We  have seen that there is no way to draw  a  straight 
    line subdividing the x-y plane so that the  exclusive-or 
    function  is represented. Unfortunately, this is not  an 
    isolated   example;  there  exists  a  large  class   of 
    functions  that cannot be represented by a  single-layer 
    network.  These  functions  are  said  to  be   linearly 
    inseparable,  and  they  set  definite  bounds  on   the 
    capabilities of single-layer networks.

    Linear  separability  limits  single-layer  networks  to 
    classification  problems  in which the  sets  of  points 
    (corresponding   to  input  values)  can  be   separated 
    geometrically. For our two-input case, the separator  is 
    a  straight  line. For three inputs, the  separation  is 
    performed by a flat plane cutting through the  resulting 
    three-dimensional  space.  For  four  or  more   inputs, 
    visualisation   breaks   down  and  we   must   mentally 
    generalise  to  a  space of n dimensions  divided  by  a 
    "hyperplane",  a  geometrical object that  subdivides  a 
    space  of  four or more dimensions.... A neuron  with  n 
    binary inputs can have 2 exp n different input patterns, 
    consisting of ones and zeros. Because each input pattern 
    can produce two different binary outputs, one and  zero, 
    there  are  2  exp  2 exp n  different  functions  of  n 
    variables.

    As  shown  [below],  the  probability  of  any  randomly 
    selected  function  being  linearly  separable   becomes 
    vanishingly   small  with  even  a  modest   number   of 
    variables. For this reason single-layer perceptrons are, 
    in practice, limited to simple problems.

   n    2 exp 2 exp n   Number of Linearly Separable Functions
   1         4                             
   2        16                             14
   3       256                            104
   4    65,536                          1,882
   5  4.3 x 10 exp  9                  94,572
   6  1.8 x 10 exp 19               5,028,134

P. D. Wasserman (1989)
Linear Separability: Ch2. Neural Computing Theory and Practice.

Hope this will suffice for now. However, as to Nets  being  restricted
to 'toy' problems, see Minsky in the 2nd edition of Perceptrons &  his
general stance in 'The Society of Mind'. My interests in this area are
the light it sheds on the  limits of  intensional judgement  (clinical
judgement vs. actuarial).

-- David Longley
