Return-Path: <@cs.cmu.edu:rodney%hsvaic.boeing.com@mcc.com> Received: from cs.cmu.edu by A.GP.CS.CMU.EDU id aa13832; 28 Jun 91 17:04:34 EDT Received: from mcc.com by CS.CMU.EDU id aa22630; 28 Jun 91 17:02:41 EDT Received: from atc.boeing.com by MCC.COM with TCP; Fri 28 Jun 91 14:58:59-CDT Received: by atc.boeing.com on Fri, 28 Jun 91 12:59:15 PDT Received: by hsvaic.boeing.com (4.1/SMI-4.1-hsvaic-s.2) id AA14751; Fri, 28 Jun 91 14:56:59 CDT Date: Fri, 28 Jun 91 14:56:59 CDT From: rodney@hsvaic.boeing.com (Rodney Daughtrey) Message-Id: <9106281956.AA14751@hsvaic.boeing.com> To: Common-Lisp@mcc.com Subject: Summary: LISP and Databases Following is a summary of responses to an information request I posted to this list about two weeks ago concerning interfacing LISP applications to databases. There seemed to be enough interest to warrant posting this (rather large) summary. The first message is the original request that I posted. Thanks again to all who responded. Rodney Daughtrey E-mail: rodney@hsvaic.boeing.com Huntsville AI Center {major site}!uw-beaver!bcsaic!hsvaic!rodney Boeing Computer Services Voice: (205)-464-4931 Fax: (205)-464-4930 ********** > Date: Mon, 10 Jun 91 11:03:14 CDT > From: rodney@hsvaic (Rodney Daughtrey) > Message-Id: <9106101603.AA13057@hsvaic.boeing.com> > To: Common-Lisp@mcc.com > Subject: LISP and Databases > Status: RO > > I would very much like to hear from anyone with experience in > interfacing a LISP application with an off-the-shelf database > (specifically ORACLE on a mainframe, but other experiences are > welcomed), in which the LISP application can read/write data from/to > the database, especially over a network. What were the seminal > issues/problems that needed to be solved? Specifically: > > 1) How was the communication managed between the LISP application and > the database? > > 2) Real world recommendations, success stories and/or horror stories > are of special interest. > > You may reply privately via e-mail unless you consider it appropriate > for the list, and if there is interest, I will summarize. > > Thanks to all who reply. > > > Rodney Daughtrey E-mail: rodney@hsvaic.boeing.com > Huntsville AI Center {major site}!uw-beaver!bcsaic!hsvaic!rodney > Boeing Computer Services Voice: (205)-464-4931 Fax: (205)-464-4930 > ***** > Date: Mon, 10 Jun 91 10:50:32 PDT > From: jaf@inference.com (Jose A. Fernandez) > Message-Id: <9106101750.AA00360@utah.inference> > Organization: Inference Corporation > Address: 550 N. Continental Blvd., El Segundo, CA 90245 > Phone: (213) 322-0200 x205 > To: rodney@hsvaic > In-Reply-To: Rodney Daughtrey's message of Mon, 10 Jun 91 11:03:14 CDT <9106101603.AA13057@hsvaic.boeing.com> > Subject: LISP and Databases > Status: RO > > Hello Rodney, > > We implemented an Oracle interface for our ART product using Sun Common Lisp. > We used SCL's foreign function interface to call from Lisp into Oracle's Pro*C > function library. This strategy let us leverage off Pro*C's automagical > networking capability (such has running the Lisp application on a Sun-4 that > talked to an Oracle SQL server installed on a Sun-3). > > The most difficult part of the exercise was controlling signal handling > behavior. SCL (really Lucid's Common Lisp implementation for the Sun) uses an > internal stack architecture (separate and apart from SunOS's default stack > architecture as seen in a.out) that one must be careful not to corrupt. The > corruption problem we had was that Oracle's Pro*C code was installing SIGCONT > and SIGIO signal handlers that were conflicting with Lisp's memory model, > resulting in the occasional and egregious "Bus trap" error. > > I hope this helps a little bit. Good luck. > ***** > Date: Mon, 10 Jun 91 16:49:57 PDT > From: yonkman@hawk.css.gov (Tom Yonkman) > Message-Id: <9106102349.AA02880@hawk.css.gov> > To: rodney@hsvaic > In-Reply-To: Rodney Daughtrey's message of Mon, 10 Jun 91 11:03:14 CDT <9106101603.AA13057@hsvaic.boeing.com> > Subject: LISP and Databases > Status: RO > > Harlequin Limited of Cambridge England has a Common Lisp which provides > several levels of access to SQL Databases (Oracle and Informix). They can > be contacted at: > > LispWorks Request > Harlequin Limited > Barington Hall > Barrington > Cambridge CB25RG > England > > Tel: +44-223-872522 > FAX: +44-223-872519 > > email: lispworks-request@harlqn.co.uk or > mcvax!ukc!lispworks-request > ***** > Date: Tue, 11 Jun 91 13:44-0000 > From: Peter Paine > Subject: LISP and Databases > To: rodney@hsvaic, Common-Lisp@mcc.com > In-Reply-To: <9106101603.AA13057@hsvaic.boeing.com> > Message-Id: <19910611134453.0.P2@porter.asl.dialnet.symbolics.com> > Status: RO > > Date: Mon, 10 Jun 91 11:03:14 CDT > From: rodney@hsvaic.boeing.com (Rodney Daughtrey) > > I would very much like to hear from anyone with experience in > interfacing a LISP application with an off-the-shelf database > (specifically ORACLE on a mainframe, ... > > I have seen a Lisp ORACLE interface advertised commercially. > > Perhaps you would get some pointers by approaching ORACLE themselves. > ***** > Date: Tue, 11 Jun 91 10:02:42 PDT > From: Siavash Hashemi (See Ya) > Subject: LISP and Databases > To: rodney@hsvaic > Status: RO > > Dear Rodney: > > We at IntelliCorp have several off-the-shelf products that can enable you > connect to a large number of databases. The product names are KEELINK and > KEECONNECTION with add on modules for different databases. > > As the names point out, you will have to use KEE with that for the off-the- > shelf state of the product. With KEELINK we do supply the source code and > you can strip off parts of the code that you don't need. > > If you are interested, I can send you literature or can arrange for a demo > for you. I know that Boing is one of our big KEE customers and I can > arrange for favorable terms for you. > > My no. is 415-965-5808, if you want to contact me for more info. > > Take care and good luck. > Sia Hashemi > KEE product manager. > ------- > ***** > Date: Wed, 12 Jun 91 06:34:14 PDT > From: jmg@inference.com (Jeff Greif) > Message-Id: <9106121334.AA15271@quaestor.Inference.COM> > Sender: jmg@inference.com > Organization: Inference Corporation > Address: 550 N. Continental Blvd., El Segundo, CA 90245 > Phone: (213) 322-0200 > To: rodney@hsvaic > Cc: jaf@inference.com > In-Reply-To: Rodney Daughtrey's message of Mon, 10 Jun 91 11:03:14 CDT <9106101603.AA13057@hsvaic.boeing.com> > Subject: LISP and Databases > Status: RO > > Inference has written a generic (pre-CLOS) CL -> SQL interface with one > complete instantiation for ORACLE and Lucid Lisp 3.0 and 4.0. This has been > used in one large application with essentially complete success. Various > generic and application-specific facilities have been provided on top for > automatically generating and executing queries to collect restricted sets of > data from the database, display menus of valid options for user input to > fields of forms, for browsing the database, etc. Currently there are no plans > to support these interfaces in our Lisp-based products, although there is a > possibility some or all of them may be issued as unsupported software as part > of a future product release. Likely the extent to which this happens will > depend upon the interest of our customers. There are no plans currently to > make the software available any other way. > > The ORACLE instantiation (the bottom layer) of the CL -> SQL interface uses > ORACLE's OCI interface functions through Lucid Lisp's foreign-function > interface to implement what is expected to be a common set of operations. > These OCI functions handle all the network communications with the ORACLE > server on the network. The foreign-function definitions are the only > Lucid-specific parts of the code. > > So far it has only been used on Suns (with database accessible via TCP) but > is fully portable to other platforms where Lucid Lisp and the ORACLE OCI are > present. It is intended to be able to support other relational databases that > accept dynamic SQL, e.g. Ingres, Sybase, but no implementation work has been > done on these. > > The problems: > > 1) Storage management. The temporal extent of the data structures created in > Lisp and passed to the database is variable. Many of these must be allocated > in static storage to prevent them from being moved by the garbage collector > while ORACLE holds their address. There are vastly different extents of the > various structures required which implies facilities are needed for handling > different varieties differently. For example, you can tell ORACLE where the > buffers are that will get the fields of a retrieved record on each record > fetch (so GC can't be called while ORACLE needs to know.) Or you can allocate > them in static space and save this not inconsiderable overhead on the > fetching. Owing to this and a general proclivity to avoid garbage > (particularly garbage in static space!), the CL -> SQL interface is > garbage-free using a simple memory-management layer. > > 2) Interrupt handling. The OCI sets up handlers for certain UNIX signals when > you first connect to it. Each time one of the OCI routines is called from > Lisp, Lisp's interrupt handlers must be swapped with the ORACLE handlers, and > swapped back on return. The set of signals whose handlers must be swapped > depend upon whether the server is on the local machine and what communications > protocol (TCP, pipes, ...) is used. There is bug-prone stuff here -- suppose > you take an interrupt or the Lisp scheduler switches tasks while the ORACLE > C handlers are in place! > > 3) ORACLE OCI documentation. This is incredibly poor -- incomplete and > riddled with errors, particularly with regard to which arguments must be > passed as arrays when the "array-processing" features of the OCI are used. > > 4) Small bugs in ORACLE OCI. There were several, including some that required > code be written to translate between Lisp floats and ORACLE internal numeric > formats, when that shouldn't have been necessary. > > 5) The generic CL -> SQL interface hands all SQL statements to the DBMS to > parse and process, and knows no SQL itself. In order to know what types will > be returned by ORACLE in the various buffers holding the values in the columns > of the retrieved records, it must make OCI calls. However, ORACLE can only > distinguish two types of data returned: numeric or character, and cannot > distinguish between floats or integers at parse time (even by consulting the > data dictionary -- consider the query > > "SELECT (CONSTANT.VALUE * CIRCLE.DIAMETER) FROM CIRCLE,CONSTANT > WHERE CONSTANT.NAME ='PI'" > > where DIAMETER contains integers). This eventually required that numeric data > be returned in ORACLE internal format, to be analyzed and converted to floats > or integers as appropriate. > ***** > Date: Fri, 14 Jun 91 16:34:37 -0500 > From: ford@m2.csc.ti.com (Steve Ford) > Message-Id: <9106142134.AA25011@m2.csc.ti.com> > To: rodney@hsvaic > Subject: Re: LISP and Databases > In-Reply-To: <1991Jun10.174702.25025@csc.ti.com> > Organization: TI Computer Science Center, Dallas > Cc: ford@csc.ti.com > Status: RO > > In article <1991Jun10.174702.25025@csc.ti.com> you write: > >I would very much like to hear from anyone with experience in > >interfacing a LISP application with an off-the-shelf database > >(specifically ORACLE on a mainframe, but other experiences are > >welcomed), in which the LISP application can read/write data from/to > >the database, especially over a network. What were the seminal > >issues/problems that needed to be solved? Specifically: > > > >1) How was the communication managed between the LISP application and > > the database? > > > >2) Real world recommendations, success stories and/or horror stories > > are of special interest. > > For the past three years, I've been doing persistence-related stuff (OODB), > mostly in Lisp, somewhat in C, and C++, in support of a large > Lisp-based VLSI CAD system with very high performance requirements. > Most of the work was on Lisp Machines, but lately I've been on a > SparcStation, mostly using Harlequin Lispworks, C and C++, RFP/XDR, > Oracle, and Informix. Some random observations: > > 1) LispWorks provides two things you might be interested in. > > a) a Lisp interface to SQL databases. I've used it to interface to > Informix. Since it basically interprets all SQL queries, its not > the fastest method we've used. > > b) a CLOS interface to the above, basically mapping a class to a relation, > an instance to a tuple, a slot to an attribute, and providing a > LISPy Object-oriented interface to data in Informix or Oracle. > Not a real smooth integration, and not too fast, but useful. > > We tried each of these out, but didn't end up using either, mostly for > performance reasons. If performance hadn't been a problem, I'd have > probably worked a little on the interface to make it more seamless with > Common Lisp. > > We didn't try using this stuff across the network, but they claimed it > worked. Basically, LispWorks loads Informix's C interface into your Lisp > image using the foreign function interface (all Lisps have these). If > you are using a networked RDB, then the RDB takes care of the network > communications, else you can just talk to a local RDB. In other words, > LispWorks is not doing any network communications itself. > > 2) After that, we built a Persistent Object Server in C/C++ which used Oracle > as a back-end, to get all the database amenities (backup, recovery, > atomicity) for free, and built our own communications facility in C, using > RPC/XDR, and building a Lisp interface to it using the Foreign Function > Interface. We also built the same thing with the Unix file system as the > backend. Our biggest problems these days are all performance related. > > a) Mapping complex OO data to a RDB results in a large storage overhead > that we hadn't experienced with earlier work on Lisp Machines, where we > had built a custom POS. For instance, a complex cyclic object graph that > required about 150K of memory required about 50 times that much space > be available in Oracle at commit time. Some of that overhead was > the result of representing objects in a RDB, the management of > interobject references, the lack of support for repeating segments in > RDBs (you need to repeat the key all over the place); some was overhead > that Oracle tacked on, and other was the result of Oracle's rollback > mechanism (everything you do during a transaction is written into the > database AND written into the rollback segment, so, in the case of abort, > the operations recorded in the rollback segment can be undone, so you > need more than twice the disk space available at commit time than you > eventually need; a shadow pages mechamism would have been much more > efficient). Since our transactions our very long, and very large > (100MB - 1GB), the space requirement is unacceptable, and we are > looking at other options, like making our file-based system more robust, > or trying out some other backends. > > b) Speed is the other problem. Since Unix Lisps are not complete (they rely > of rather clumsy (not seamless - I can't use aref on a C string) > interfaces to Unix/C to provide much of their services (network > interfaces), we've found it difficult to implement our system in > nice clean modular portable code, isolating the implementation-dependent > C-interface stuff, without introducing too much copying. Stuff gets > copied from memory into a lisp string, which is then copied into a C > string, which then gets copied across the network by RPC, where it > gets copied into another C string, and then Oracle eventually gets it > and does who knows what, but it, at least, copies across process > boundaries. This is all so much worse than Lisp Machines, where > everything is in the same address space, and I just copied from memory > into a disk buffer, which was then written to disk. > > Overall, I'm rather disappointed that the shortcut I thought I was taking > was a deadend. However, not too many people I know have my kind of > performance requirements. LispWorks stuff might work for you, or you might > check with Andreas Paepke at HP Labs, who has implemented persistence for > CLOS and interfaced that to a number of databases. > > Have fun, > > Steve Ford > Computer Science Center > Texas Instruments > Net: ford@csc.ti.com > Tel: (214) 917-7432 > ***** > From: Cris Kobryn > Date: Mon, 17 Jun 91 14:58:28 BST > Message-Id: <2992.9106171358@watson.harlqn.co.uk> > To: rodney@hsvaic > Cc: Common-Lisp@mcc.com > In-Reply-To: Rodney Daughtrey's message of Mon, 10 Jun 91 11:03:14 CDT <9106101603.AA13057@hsvaic.boeing.com> > Subject: LISP and Databases > Status: RO > > > > I would very much like to hear from anyone with experience in > interfacing a LISP application with an off-the-shelf database > (specifically ORACLE on a mainframe, but other experiences are > welcomed), in which the LISP application can read/write data from/to > the database, especially over a network. > . . . > > You may be interested to know that Harlequin Limited has two > Lisp-based software packages which address the problems you > describe: > > o LispWorks SQL interface > -- provides functional Lisp and a CLOS SQL interfaces > to RDBMS's for data retrieval and update. Remote > networked RDBMS's are supported. > o DataWorks > -- provides a GUI toolkit for the LispWorks SQL interface. > Data may be displayed using icons, images and graphic > layouts. > > The LispWorks SQL interface is part of Harlequin's LispWorks, a > comprehensive Common Lisp programming environment for UNIX > platforms. The LispWorks SQL interface and DataWorks currently > support the following RDBMS products: Oracle, Ingres, and Informix. > Interfaces to Sybase and Unify are planned. > > If you require further technical information re these products, or > would like to communicate with end users, you should contact > Helen Goddard in our marketing department: > > Helen Goddard Email: helen@uk.co.harlqn > Symbolic Processing Marketing Phone: +44 223 872522 > Harlequin Limited Fax: +44 223 872519 > Barrington Hall > Barrington > Cambridge CB2 5RG > > I hope this helps. > > -- Cris Kobryn > ***** > Date: 14 Jun 91 16:36 -0700 > To: steve@hsvaic > In-Reply-To: <320@hsvaic.boeing.com> > Message-Id: <9106142336.AA14696@acrab.cs.sfu.ca> > Subject: Re: LISP and Databases > Status: RO > > > Below is the top level function that interfaces between our NLI "SystemX" > and a Pro*C executable "SOI" that accepts dynamic SQL (See appendix E > in Version 1.1 Pro*C manual). > > > ;;; -*- Mode:LISP; Readtable:CL -*- > > ;;;; This file contains the LISP functions which communicate with SystemX-Oracle-Interface > ;;;; (SOI) SQL Executor. The function CONSULT is called at the highest level, starts up > ;;;; SOI, solicits (SQL) input for SOI via SystemX, and prints the results. > > ;;;; CONSULT > ;;;; / \ sql > ;;;; LISP sql / +---------------------- > ;;;; / | \ > ;;;; SYSX | SOI > ;;;; english / | \ UNIX/C > ;;;; ++++++++++++++++++++++ \ sql > ;;;; / + \ > ;;;; REALITY USER + ORACLE > > (defvar *sx-ora-interface* ") > > > ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; CONSULT ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; > > ;;; INPUT: nil > ;;; OUTPUT: irrelevent > > (defun consult (&aux err-exit?) > "Interactively query the Oracle database in English." > (multiple-value-bind (stream err-stream arg3 pid) > (run-program *sx-ora-interface* > :input :stream > :output :stream > :error-output :stream > :wait nil) > (declare (ignore arg3 pid)) > (sysx-condition consult > (with-open-stream (unix stream) > (when (login-oracle unix) > (unwind-protect > (do ((sql nil) > (more? t)) ; do until > ((or (null more?) ; no more queries, or > err-exit?)) ; SOI killed due to Oracle error > (if (oracle-err err-stream) > (setq err-exit? (seek-continuation unix)) > (multiple-value-setq (more? sql) (query-oracle unix))) > (when more? (display-results unix more?))) > (if (not err-exit?) ; ie. if SOI still running > (send-input unix "/exit;"))))) > (quit (format t "~%Exiting SystemX"))))) > > > -- > Gary Hall | Voice (604) 291-3208 | INTERNET: hall@cs.sfu.ca > Centre for Systems Science | Fax (604) 291-4951 | BITNET: hall%cs.sfu.ca@ > Simon Fraser University | | relay.cdnnet.ca > Burnaby, B.C. V5A 1S6 | > > ***** > Date: Mon, 17 Jun 91 10:39:37 -0500 > From: dcmartin@cs.wisc.edu (David C. Martin) > Message-Id: <9106171539.AA24685@wilma.cs.wisc.edu> > Received: by wilma.cs.wisc.edu; Mon, 17 Jun 91 10:39:37 -0500 > To: steve@hsvaic > Subject: Re: LISP and Databases > Newsgroups: comp.databases > In-Reply-To: <320@hsvaic.boeing.com> > Organization: University of Wisconsin - Madison > Cc: > Status: RO > > You probably want to talk to either John Irwin @ Franz Inc (jdi@franz.com) > or Larry Rowe (rowe@postgres.berkeley.edu) about CLING (Common LISP Interface > to Ingres). > > dcm > -- > ----- > David C. Martin arpa: dcmartin@cs.wisc.edu > University of Wisconsin - Madison uucp: uunet!ucbarpa!dcmartin > Computer Sciences Department at&t: 608/262-6624 (O) > > ***** > Date: Fri, 14 Jun 91 13:56:10 CDT > From: us267384@serc.3m.com (Steven M. Anastasi) > Message-Id: <9106141856.AA14921@mmm.serc.3m.com> > To: steve@hsvaic > Subject: Re: LISP and Databases > Status: RO > > > I would very much like to hear from anyone with experience in > > interfacing a LISP application with an off-the-shelf database > > (specifically ORACLE on a mainframe, but other experiences are > > welcomed), in which the LISP application can read/write data from/to > > the database, especially over a network. What were the seminal > > issues/problems that needed to be solved? Specifically: > > > If you're seroius about lisp applications and persistiency, look into > Itasca Systems. They have a good, truely distributed database system > that sits on top of the lisp engine. The data representation scheme > they use is CLOS-like. > ***** > Date: Thu, 20 Jun 91 22:29:21 +0200 > From: thomas%INFORMATIK.MATHEMATIK.UNI-MAINZ.DE@CUNYVM.CUNY.EDU (Thomas Schueler) > Subject: Re: LISP and ORACLE > To: steve@HSVAIC > Message-Id: <9106202029.AA03946@Informatik.Mathematik.Uni-Mainz.DE> > X-Envelope-To: steve@HSVAIC.BOEING.COM > Status: RO > > May be you can specify what LISP and what OS you are using. In general I > would say that whenever you are able to call C ,or an other language wich > is able to use the OCI, from LISP there should be no problem. > (If you want to hear horror stories ask me about PROLOG and ORACLE ). > > regards, > thomas >