Newsgroups: comp.ai.doc-analysis.misc,comp.ai.nat-lang
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!newsfeed.pitt.edu!gatech!news.mathworks.com!uunet!in2.uu.net!bcstec!nntp
From: Richard Wojcik <rwojcik@redwood.rt.cs.boeing.com>
Subject: Re: Software Knows to Deconstruct in Plain English
X-Nntp-Posting-Host: snake.rt.cs.boeing.com
Content-Type: text/plain; charset=us-ascii
Message-ID: <3131EC75.214A@redwood.rt.cs.boeing.com>
Sender: nntp@bcstec.ca.boeing.com (NNTP News Access)
Content-Transfer-Encoding: 7bit
Organization: Research & Technology
References: <4ftflg$st9@la1.digilink.net> <4fvchh$bcl@thor.cs.cornell.edu>
		<4g3656$gpt@la1.digilink.net>
		<TED.96Feb18222415@hellespont.crl.nmsu.edu>
		<4gdgnn$d2h@la1.digilink.net>
		<DOWDING.96Feb20182843@Gansett.ai.sri.com>
		<312B54B2.28DA@redwood.rt.cs.boeing.com> <DOWDING.96Feb24104925@Gansett.ai.sri.com>
Mime-Version: 1.0
Date: Mon, 26 Feb 1996 17:23:01 GMT
X-Mailer: Mozilla 2.0 (X11; I; SunOS 5.4 sun4c)
Lines: 61
Xref: glinda.oz.cs.cmu.edu comp.ai.doc-analysis.misc:157 comp.ai.nat-lang:4612

John Dowding wrote:
> 
> >>>>> "Richard" == Richard Wojcik <rwojcik@redwood.rt.cs.boeing.com> writes:
> 
>     Richard> Does the program fail
>     Richard> because it can't make sense of every question?
> 
> Of course not, isn't this exactly why we run through exercises like
> MUC, to give us a chance to evaluate our systems in a blind test
> against other systems?

That's a tough one to answer.  I have always felt that MUC invited the
results it got by pushing the envelope too far too soon.  I agree with
you that participation in such exercises is good, and I am not defending
Inquizit (or attacking it) for failure to participate.  I have a feeling
that the marketplace is going to be a much more difficult, and ultimately
more meaningful, test of the program.  MUC-like exercises don't necessarily
address the same concerns that real customers have.  Let me remind you of
an amusing observation that Yorick Wilks once made about the success of
that old dinosaur SYSTRAN:  "There is a tremendous market out
there for bad translations."  He got a laugh from his audience, but his
serious point was that researcher perceptions and customer perceptions are
not always the same.

>     Richard> Perhaps
>     Richard> Inquizit needs to fail more gracefully than it does when
>     Richard> it runs into a brick wall of a sentence.  The true test,
>     Richard> however, is whether it responds to the needs of users who
>     Richard> understand the database and what information they need to
>     Richard> get out of it. Those users are likely to make fewer
>     Richard> goofball queries and also likely to be more forgiving of
>     Richard> occasional goofball responses.
> 
> I think that you're being a little bit hard on the questioner here.  I
> didn't think the question that was asked was at all 'goofball', it was
> just outside of the capability of Inquizit's system.  An appropriate
> response from Inquizit would be that they answered N% of queries in a
> blind test correctly, not to question the motives of the asker.

Perhaps I shouldn't have used the loaded word 'goofball', but I don't
really think that the query at issue here was reasonable.  The question
is how well the system does what it is supposed to do, not whether it
answers N% of anything correctly.  It is entirely possible that you will
get different performance from different populations of askers.  All I
said was that it had to satisfy real customers, not just anyone who can
think up a question to ask it.

> For my 2 cents, making unsupported claims like theirs in a newsgroup
> like comp.ai.nat-lang is just begging people to push the envelope.

The harder they beg, the more you must try to resist them.  ;-)  We all
know that this medium can cause people to abuse each other more than
they normally would in face-to-face situations.  We should ignore the
provocations and concentrate on getting at the truth. 

-- 
Richard H. Wojcik, Natural Language Processing
Boeing Information & Support Services
P.O. Box 3707, MS 7L-43, Seattle, WA 98124-2207 (phone: 206-865-3844)
rwojcik@redwood.rt.cs.boeing.com                (fax:   206-865-2965)
----Opinions expressed above are not those of The Boeing Company----
