Newsgroups: comp.ai.neural-nets
Path: cantaloupe.srv.cs.cmu.edu!nntp.club.cc.cmu.edu!eecs-usenet-02.mit.edu!nntprelay.mathworks.com!howland.erols.net!ix.netcom.com!hoang
From: hoang@netcom.com (ah)
Subject: Re: Some questions about Cross entropy error function
Message-ID: <hoangEDHxG5.4v5@netcom.com>
Organization: Netcom On-Line Services
X-Newsreader: TIN [version 1.2 PL2]
References: <33C7B1CB.7E01@musc.edu>
Date: Fri, 18 Jul 1997 03:52:53 GMT
Lines: 43
Sender: hoang@netcom5.netcom.com



Hi, The cross entropy error function as defined 'G. E. Hinton, 
Connectionist learning procedures, Artificial Intelligence, 
Vol. 40, 1989' is just another name for the minus of the average 
of the log likelihood of the data, given that the output vectors 
can assume only values 0 or 1. A practical example is as follows: 
Let the input units represent customer data like income, age,.. and a 
single output unit represents whether or not (1/0) the customer 
responds to some previous direct-mailing program; and we (American
Express for instance) are interested in estimating for any arbitrary
customer what the probability is (s)he will respond to a future mass 
direct-mailing program. (Similar interpretation for survival outcomes).

A slight variation of the above cross entropy error function is the 
minus of the log of the likelihhood of the data. There are other 
variations as well, the usefulness of these variations will depend 
on their applicability and appropriateness for the problem at hand. 
The name cross entropy is also given to a variation of the 
Kullback-Leibler information function, which is totally different 
from the likelihood function.

Albert


--------------------------


qin pan (panq@musc.edu) wrote:
: Hello,

: Regarding to cross entropy error function in training MLP, I found
: several different formulas in different articles. I have no idea which
: one is correct. Does anyone have any experience with this error 
: function (like which formula and its derivative, the coding of outputs
: 0/1 or 0.1/0.9, etc)?

: Any help appreciated.

: -- Lou

: Department of Biometry and Epidemiology
: Medical Univ of South Carolina
