Newsgroups: comp.ai.doc-analysis.misc,comp.ai.nat-lang
Path: cantaloupe.srv.cs.cmu.edu!bb3.andrew.cmu.edu!newsfeed.pitt.edu!gatech!swrinde!newsfeed.internetmci.com!in1.uu.net!bcstec!nntp
From: Richard Wojcik <rwojcik@redwood.rt.cs.boeing.com>
Subject: Re: Software Knows to Deconstruct in Plain English
X-Nntp-Posting-Host: snake.rt.cs.boeing.com
Content-Type: text/plain; charset=us-ascii
Message-ID: <312E0FE2.65D4@redwood.rt.cs.boeing.com>
Sender: nntp@bcstec.ca.boeing.com (NNTP News Access)
Content-Transfer-Encoding: 7bit
Organization: Research & Technology
References: <4ftflg$st9@la1.digilink.net> <4fvchh$bcl@thor.cs.cornell.edu> <4g3656$gpt@la1.digilink.net> <TED.96Feb18222415@hellespont.crl.nmsu.edu> <4gdg <4gj1nt$pc2@la1.digilink.net> <4gjuh5$m2t@anarchy.io.com>
Mime-Version: 1.0
Date: Fri, 23 Feb 1996 19:05:06 GMT
X-Mailer: Mozilla 2.0 (X11; I; SunOS 5.4 sun4c)
Lines: 52
Xref: glinda.oz.cs.cmu.edu comp.ai.doc-analysis.misc:144 comp.ai.nat-lang:4595

I have liked Jonathon Slocum's approach to testing Inquizit best, so
far.  He has actually uncovered some useful information that might
help the developers improve their product.  

All of this reminds me of an NLI product that came out 9 years ago--
NLI Datatalker.  It happened to be my first assignment at Boeing to
go down to San Francisco and evaluate the product.  One problem with
that company was that it made similar hyperbolic claims about its
product, and it even had the effrontery to copyright the acronym "NLI",
which everybody in the industry was using as a generic term.  Their
product was really pretty good for what it was--an NL interface to
a relational database.  Unfortunately, the company went to unnecessary
lengths to make NLP researchers upset with it.  (The company president
attended a workshop panel once at a conference, and I remember him
nimbly vaulting a table to reach a side entrance at the end of the panel 
session.  Had a plane to catch, I guess.  :-)  Marketing hype may
seem like a necessary evil when you are trying to sell a product or
sell stock, but you really need to think about how it will be received
in the NLP community.  Above all, avoid the words "Holy Grail".  :^)

Let's not forget that Inquizit is not an NLI to a real database.  It
just retrieves documents.  It can't really be expected to reason very
deeply about the relevance of the documents to a question, particularly
a question that requires abstraction from content.  So the best way to
evaluate it is to see how much better or worse it does than other 
document retrieval systems in coming back with relevant hits. 

The web demo is a very nice thing to have, and I applaud the company
for having the courage to put it out there.  For all of the irrelevant
and angry responses they are getting, they also seem to be getting 
some data on the robustness and accuracy of the system.  It really
is important that the system be able to answer questions about itself.
It needs to recover more gracefully (and quickly) when it is sent on
fruitless missions or doesn't have a well-formed input.  It is probably
not a good idea to force people to type fully grammatical queries, since
telegraphic ones are more natural.  Maybe the product should put out a 
paraphrase of the query just to get user confirmation before going out 
on a search.  I would certainly like to see the system operate over a 
much larger set of documents, and I hope that the slowness of some
responses can be traced to ephemeral hardware or network problems.

Marketing hype aside, I wish ITP all the success in the world.  Every
successful NLP product is a success for our field.  Criticism should
be honest and candid, but it costs nothing to deliver it with 
professionalism and courtesy.

-- 
Richard H. Wojcik, Natural Language Processing
Boeing Information & Support Services
P.O. Box 3707, MS 7L-43, Seattle, WA 98124-2207 (phone: 206-865-3844)
rwojcik@redwood.rt.cs.boeing.com                (fax:   206-865-2965)
----Opinions expressed above are not those of The Boeing Company----
