From newshub.ccs.yorku.ca!torn!cs.utexas.edu!sun-barr!olivea!uunet!elroy.jpl.nasa.gov!usc!rpi!bu.edu!buphy0.bu.edu!leao Fri Sep  4 09:41:02 EDT 1992
Article 6699 of comp.ai.philosophy:
Path: newshub.ccs.yorku.ca!torn!cs.utexas.edu!sun-barr!olivea!uunet!elroy.jpl.nasa.gov!usc!rpi!bu.edu!buphy0.bu.edu!leao
>From: leao@buphy0.bu.edu (Joao Leao)
Newsgroups: comp.ai.philosophy
Subject: Re: Loebner prize
Message-ID: <94511@bu.edu>
Date: 25 Aug 92 18:33:10 GMT
References: <BILL.92Aug25000110@ca3.nsma.arizona.edu>
Sender: news@bu.edu
Lines: 86

In article <BILL.92Aug25000110@ca3.nsma.arizona.edu>, bill@nsma.arizona.edu (Bill Skaggs) writes:
|> Thanks, Joao, for posting the transcripts.  Comments:

Thanks for your notes. I will try to respond with what I gather at the
competition. I should probably explain that I attended as a member of the
press (I wrote an article on it for a portuguese magazine.)

|> 
|> 1)  The performance of the machines was quite disappointing.  None of
|> them seemed any more responsive than Eliza -- most less.  It seems
|> pretty obvious that these programs were not taken from the top rank of
|> state-of-the-art natural language systems.
|> 

Indeed! At the competition Dennett and Epstein apologized profusely for the
low quality of the competitors. Their idea was just to "start the ball rolling" on an appeal to empirical implementation of the Turing Test idea, which they
believe is close to Turing's original intention. The prize of $1,000 could not
pay for anything better than what people had already put together mostly in
amateuristic attempts to mimic Eliza/Parry style "dialoguers".
Weizenbaum agreed that none of the entries adds anything new to the basic Eliza premiss. The hope is that future editions of Loebner will bring ou the witts
of the porfessionals. That was what I heard from Dennett... 

|> 2)  Most of the judges were hapless.  They actually permitted the
|> competitors to ask *them* questions!  I wonder if it would violate the
|> spirit of the Turing Test to first give the judge a few pointers about
|> elementary interviewing techniques?  :-)
|> 

That is also quite clear from the transcripts. The people who were hired as
judges were selected for their computer "unsaviness" or haplessness. I guess
they did a swell job on this account! I tend to agree that these people should
have been "rehearsed" in some way, though I understand that is a controversial 
topic among the Loebner Committee people. Instead they were considerably 
hindered by the referees who might have shortened the dialogues with their
interventions.

In any case the results surprised everyone! As you probably know the winning program got the "human" flag from 5 out of the 10 judges! Please remember that Turing's prediction was 75% by 1999 (on 5min rather than 8min dialogues).
Given the notorious incompetence of the programs who can you blame for such
result? Because I believe that the TT is as much a test of human stupidity
as of machine intelligence and I believe that Turing proposed it in that
spirit I am somewhat relieved that the match up started with even witts...

But I am sure that the people who are doing it this year will pay attention
to that aspect of the test and we will see better programs and better judges
hopefully. The results were also embarrassing, I am sure...

|> 3)  It seems a bit unfair to pick a human competitor with as much wit
|> as Terminal 4.  He's certainly a more entertaining than average
|> conversationalist. 

Well, Terminal 4 was judged a machine by 3 of the 10, as I have heard it!
Funny that you mention it because there was a little debate in the audience
about it because as you know Shakespeare has been on disk for a long while
and you and I know that would be relatively easy to put together a program
that would query the works of the bard for something apropos...

As it turns out that was the intended effect by the people in the committee.
By the way, in one of the few interesting pieces of the dialogue, in which
you see judge 9, craftier than most, violate the fixed context rule without
the referee's intervention:
...
01:26:29  Judge 9
I don't know if you follow politics, but recently they have been referring to M
ario Cuomo as Hamlet on the Hudson, referring to his indesicion.  Can you expan
d on this analogy.

01:27:33  Terminal 4
His brooding.  Getting to the sonnets, I really like the one that goes
When in disgrace with fortune and men's eyes/I all alone beweep my outcast
state....
...

you realize at an instant what is a "killer question" really is and what
the frame problem is all about even if you were never exposed to any thing
like this. 
Anyway, I think I will post the terminal IDs now...

|> 
|> 	-- Bill

-- 
Joao Pedro Leao (Artificial Iconoclast and Director of Computer Resources
Artificial Physics Lab * Boston University - Physics Dept. Boston MA 02215)
 	leao@buphy.bu.edu | leao@buphyc.bitnet | BUPHYC::LEAO 
"Well I am sitting here in Tahiti/ I am laying in the sun and sipping a...
...chartreuse tropical drink!/ and I say: I know those Bermuda shorts!..."


