Newsgroups: comp.ai
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!news3.near.net!noc.near.net!paperboy.wellfleet.com!news-feed-1.peachnet.edu!gatech!newsxfer.itd.umich.edu!zip.eecs.umich.edu!panix!news.mathworks.com!news.alpha.net!uwm.edu!spool.mu.edu!torn!nott!dgbt!rick.dgbt.doc.ca!thom
From: thom@rick.dgbt.doc.ca (Thom Whalen)
Subject: Re: Why I Like the Loebner Competition
Message-ID: <1995Mar14.171659.23843@dgbt.doc.ca>
Sender: news@dgbt.doc.ca (News user)
Organization: Communications Research Centre, Ottawa, Canada
References: <1995Mar8.162130.10752@dgbt.doc.ca> <fano-1103951313070001@fano.ils.nwu.edu> <1995Mar12.224351.7345@dgbt.doc.ca> <fano-1303951110030001@fano.ils.nwu.edu>
Date: Tue, 14 Mar 95 17:16:59 GMT
Lines: 64

> > This was necessary for the objective of my research, which is not to
> > simulate intelligence, but merely to make it easy for people to get
> > information from a computer. 

> That's a laudable goal, but what's the point of subjecting such a system
> to the Turing Test?
> What would it prove one way or the other?

What if it works?  Joy.  Occam's Razor is satisfied.  We have a simpler
theory of natural language than those proposed by computational
linguists.

What if (more likely) it does not work?  Then we can look at the
reasons why.  We learned a lot from the failures of Winograd's SHURDLU
program and from the failures of Weizenbaum's Eliza.  Linguists learn
from the exceptions that cause their theories to stumble.  I have
learned something every time one of my programs failed in the past and
I would expect to learn something from the failure of TIPS to pass a
Turing test.

In fact, I predicted poor performance from TIPS in the last Loebner
contest because of cultural differences between the judges (who were
journalists) and the target audience who were Internet surfers.  As
well, I predicted poor performance because it was tested in a very
public forum with a socially sensitive topic when it had been developed
from private conversations.

My predictions were confirmed and the underlying socio-linguistic
hypotheses supported.  TIPS/sex responds appropriately to about 80 per
cent of the queries that it receives on the Internet, but only
responded appropriately to 50 per cent of the queries received during
the competition.  Furthermore, as predicted, the questions submitted by
the judges were longer, less direct, and more on the periphery of the
topic.

What I did not predict was that, despite it's poor performance, it was
still judged as (marginally) more human-like than the other
competitors.  I fully expected to lose the contest.  That I did not was
a happy bonus.  I expect that next time, a program based on one of
these "sophisticated" AI theories will blow TIPS away.  But, then, my
expectations have been dashed before.

The more difficult question implied by yours is, "Given that no one has
passed the Turing test yet, whose approach will be the most likely to
provide the next step on the path to eventually passing a Turing
test?"  Much of the discussion that I have heard (from the AI
establishment) assumes that computational linguistics is obviously the
right direction and that alternative, behaviourally-oriented approaches
are abberations which distract people from the true path.  I think that
this is a very tenuous assumption, to say the least.

I have not yet seen sufficient concrete evidence of the success of
computational linguistics to convince me to abandon my present course
and start staying up all night reading their journals.

If you want to see concrete evidence that "computational behaviourism"
has some merit:
	telnet debra.dgbt.doc.ca 3000
and ask about sex.

-- 
   Thomas Whalen, Ph.D.   thom@dgbt.doc.ca   (613) 990-4683       
     Communications Research Centre, Government of Canada
         3710 Carling Avenue, Ottawa, Canada  K2H 8S2
