\begindata{text,539955892} \textdsversion{12} \template{default} \define{symbola attr:[FontFamily SymbolA Int 0]} \define{footnote } \formatnote{.ds CT .ds CB - V-\\\\n(PN -}\majorheading{ A Guided Tour of \typewriter{.../contrib/mit} }\center{ By Bill Cattey MIT Information Systems Last Updated: \begindata{writestamp,539959560} Datastream version: 1 %o %A, %Y andysans12 717969942 \enddata{writestamp,539959560} \view{writestampview,539959560,96,0,0} } Welcome to the wonderful world of MIT-authored software for use with ATK. Alas, it is a rather chaotic world at the present time. This paper should help you bring order out of this chaos and to make this snapshot of what we use useful at your local site. If you've read the README.ez and WHAT_IS_NEOS files in the distribution directory, you will be familiar with some of this material. I intend to give a more detailed map of the territory without putting you to sleep. I hope to give answers to these questions: What files are there? What helpful functionality is there? What works? How well? How hard is it to make it work in different environments? What is on the TODO list for future functionality or bug fixes? Where did these functions come from? What new functions might we expect in the future? \section{The Directories } Instead of putting an ls -lR here, (which you can do yourself) I would like to call your attention to important files and directories, and how to group them in your mind as you decide what you want to use from here. Details of function, status and history appear later. The \typewriter{contrib/mit} directory contains the following subdirectories: \leftindent{\typewriter{annot} -- Two insets: one for annotating and one for holding PostScript \typewriter{util} -- miscellaneous utilities discussed in detail below \typewriter{scribetext} -- translates Scribe format files to ATK datastream \typewriter{2rtf} -- translates ATK datastream to Microsoft Rich Text Format \typewriter{rtf2} -- translates from Microsoft Rich Text Format to ATK datastream \typewriter{neos} -- classroom oriented file exchange applications based on ez \typewriter{fxlib} -- file exchange libraries used by neos \typewriter{com_err} -- a copy of the Athena common error package (used by fxlib) } For the most part an exhaustive description of every file in every directory makes no sense. Such a dialectic would only be of interest to developers intent on making non-trivial modifications to the various systems. However, a nearly file-by-file look at \typewriter{annot} and \typewriter{util} does make sense. \section{The annot and util directories } In \typewriter{util} we have: \example{\smaller{Imakefile ez2ascii.c header.c headrtv.c vutils.c RCS ez2ascii.help header.ch headrtv.ch vutils.ch compat.c ez2ps header.doc popts.c compat.ch ez2ps.help headertxt.help popts.ch }} What are these? \leftindent{\description{\bold{ez2ascii} converts ATK datastream to plain ASCII for display on character terminals. Uses the same algorithm as messages. \bold{ez2ps} is a shell script that makes PostScript documents from ATK documents. \bold{header} is an inset to put running headers and/or footers into a document. \bold{popts} is an object used to set printer options in a dialogue box. \bold{vutils} is a package of utility functions to hang off any view as either a key or a menu command. Currently consists only of the vutils-fork-helpproc command which starts andrew help on command from any view. \bold{compat }is a package of commands in use at MIT for improved compatibility between ez and GNU Emacs. }} In \typewriter{annot} is an \bold{icon} object which is a base class for two objects, \bold{note}, and \bold{ps}. Each subclass of \bold{icon} contained in \typewriter{annot} is a view with two modes of display: closed it looks like an icon of a sheet of paper. Open, it is a scrolling text object. Each is intended as a wrapper for different kinds of command text. Each has a unique print method. The note object prints as a footnote. In fact, it can be considered a different user interface to the existing footnote system. The \bold{ps} object contains PostScript text which is correctly rendered by transcript troff at print time. The latest version contains code from Mike McInerny which can render the PostScript if your X server has the PostScript extension. A new subclass of icon is under construction which will hold troff commands. It is intended as an alternative to the Format Note style in running text. \bold{ez2ascii} and \bold{ez2ps} are simply shell commands. They can also be used as filters taking input from or sending output to a UNIX pipe. The \bold{compat} package is an ongoing effort and will be discussed in greater detail below in the GNU Emacs Compatibility section. The \bold{header} inset is intended as a friendly user interface to troff commands that print three part headers and footers on every page. Like the subclasses of \bold{icon} it has an open and closed mode of display. When open it is a small rectangle about 20 pixels high that tells whether the object is describing running heads or feet. When open, there is a box to contain the contents of the left, middle, and right portions of a troff three part header or footer. Following the ATK tradition of simple general insets reused in multiple contexts, the \bold{header} inset does not try to do both header and footer at the same time, nor does it try to set positions in the document. It is a unification of the MacWrite ruler concept and the ATK inset. To define both headers and footers, insert two \bold{header} objects and set the "footer" mode in one of them. Place them in your document wherever you want to set new headers and footers. Note that the header & footer commands affect only subsequent pages - the header will appear at the top of the next following page; the footer will appear at the foot of the page into which it is inserted. \section{GNU Emacs Compatibility } At MIT the most popular editor (as one might expect) is GNU Emacs. A marketing decision I made when first deploying ez to the Athena computing environment was to say, "If you find a difference, any difference, between ez and GNU Emacs, it is a bug! Report it. I will fix it." This helped users get over the shock of a command set that wasn't exactly what they were used to, and saved me some of the work of trying out all the commands. Eventually we hired a student who did a command-by-command comparison of ez and GNU Emacs. Although it was obvious to me that the ATK community would be interested in GNU Emacs compatibility, I somehow never made the MIT compatibility stuff into a distribution that one could just \typewriter{make dependInstall}. Well, guys, I finally got the message and hope to have something by the end of the summer. The current state of GNU Emacs compatibility is a bit of a mess: \symbola{7} There are two files \typewriter{WallChart.ez} and \typewriter{RefCard.ez} which name all the commands we thought were interesting to both users of ez and GNU Emacs, and the differences between the files. Alas, these files report differences we now hide. \symbola{7} There is a gnu.init file containing key bindings that use the GNU Emacs keys for commands where they differ from those of ez. \symbola{7} We have a local ezkeys.help file which documents these key bindings. \symbola{7} Buried in \typewriter{atk/text/txtvcmod.c} is t\typewriter{extview-MIT-kill-line} which consolidates line kills into the kill ring as GNU emacs does, and word kill commands which puts deleted words onto the kill ring. \symbola{7} We have a package, gnucompat, which was intended to contain GNU Emacs compatibility functions, but ended up as a catchall for new commands that the MIT market demanded: \description{\leftindent{gnucompat-fill-para Fill Paragraph. gnucompat-insert-bullets Put a bullet at the beginning of each paragraph in the region. gnucompat-remove-bullets Remove bullets from the beginning of each paragraph in the region. gnucompat-enumerate Put number in sequence at the beginning of each paragraph in the region. gnucompat-denumerate Delete number from the beginning of each paragraph in the region. gnucompat-insert-inset Insert a named inset here. gnucompat-insert-inset-no-nl Insert a named inset without newlines around it. gnucompat-add-default-tpl Enable basic styles by adding the template named default. }} The only command here truly relevant to GNU Emacs compatibility is Fill Paragraph which we bind to -q so that people in ez will get the same effect of a paragraph fill as they would in GNU Emacs. The GNU Emacs compatibility required cooperation between CMU and MIT: Several years ago the ATK developers added a facility for a global init file that would always be read, even if a user had their own init file. Nobody understood why I required this until I said, "I have GNU Emacs key bindings that I want everybody to get. They must get them even if they have their own init files, without having to remember to add a line to include my init file." The line and word kill commands required either adding MIT's local hacks to the textview object, or adding more interface routines to expose the insides of the text object. Bob Glickstein worked very hard to make the gsearch package bit for bit compatible with GNU Emacs even when I pointed out a nasty change that had to be made to the way the interaction manager handled key processing. Here is an example of differences we plan to hide by adding new textview commands -- the set of commands that capitalize and lower case words: \leftindent{\heading{Case Change:} \typewriter{M-u M-u upper case word} \typewriter{M-l M-l lower case word} \typewriter{M-c M-c capitalize word} When changing the case of words, GNU Emacs finds the first word on or after the cursor, looking past spaces, tabs, and newlines when necessary, operates on that word, and moves the cursor immediately past the word. Ez operates on a word only if the cursor is on the word or immediately after it, and does not move the cursor. } The new upper case word command would use the GNU Emacs heuristic for finding a word, and would move the cursor to the end of it. The new command would not be the default, since that would disorient the present users of ez. As with the word kill commands, it would be enabled through gnu.init key binding setting at run time. Our plan for the future is to add more textview commands, document all the changes\footnote{\ \begindata{fnote,539679752} \textdsversion{12} An important topic in the document would be discussion incompatiblities left in because the ez behavior is more correct in the particular context. For examplem, not warping the cursor to the middle of the screen on issuance of the ^V scrolling command because ez has the additional feature of a scroll position independent of the character insertion point.\ \enddata{fnote,539679752} \view{fnotev,539679752,97,0,0}}, and move many of the scattered pieces of GNU Emacs compatibility to a single place. We have a student programmer working on this task even now. \section{Scribe to ATK translation } The \typewriter{scribetext} directory contains a useful translator that takes a Scribe document and tries to convert it to an ATK datastream. The purpose of \bold{scribetext} was to provide the MIT users of the Scribe text formatter incentive to convert to using EZ. The program never achieved popularity. Reasons for this follow. Certain commonly used Scribe commands were not well handled. There was no ATK to Scribe conversion utility for those who changed their minds. The program was not well marketed. The program contained a few rough edges that were never smoothed off. The program is designed to be table driven.\footnote{\ \begindata{fnote,539680008} \textdsversion{12} I've just discovered that scribetext.alg, the file that tells how to use the table was mutilated at some point. A subsequent patch will contain the correct file. Until then, the .alg files in rtf2 or 2rtf cover the same material. (The two RTF programs are based on scribetext.)\ \enddata{fnote,539680008} \view{fnotev,539680008,98,0,0}} Here is what you need to know to understand the translation table: The transfile consists of lines with two strings per line, the first string known as the scribe command (sometimes referred to as the ".scribe" entry) and the second string known as the ez command (also the ".ez" entry). Scribe commands matching the first string are converted to the second string. If the second string begins with one of the following special characters: '@', '#', or '$' string-to-string conversion is not performed. Instead, a hard-coded C procedure is called with the given name. For example, ATK does not have a comment style. Thus, Scribe comments are deleted from the output by this directive in the scribe.trans file: \example{comment @delete } Interface could be made to a future ATK feature by changing the @delete command designation to a C procedure coded into a subsequent version of scribetext. One extremely inconvenient architectural bug in the parser is that it does not deal with scribe commands that can appear both with and without arguments. So the @newpage command is not properly parsed unless it is followed by an argument: \leftindent{@newpage(3) works @newpage() works @newpage does not work } If you can deal with editing your @newpage commands, the rest of the translation is very intelligent. It is comparatively easy to add new functions for currently unsupported Scribe keywords. To enable compilation of \bold{scribetext} in your ATK build, set \typewriter{SCRIBETEXT_ENV = 1} in your site.h file. \section{Translation between ATK and RTF } The \bold{scribetext} program served as the basis for \bold{rtf2} and \bold{2rtf}. Scott Rixner wrote the two programs and then left the project without fully debugging them. This is a fact of life when one hires student programmers for one term. The \bold{rtf2} and \bold{2rtf} programs contain known bugs which are documented in the TODO list. The programs are being distributed in their broken state in the hope that they will be useful to people, and that a simple patch will be able to make them work correctly in the future. There is also the hope that someone else will find the bugs and fix them so that the beleagured maintainer of ATK at MIT will be spared the task. An ambitious level of functionality was set for these programs: \description{\symbola{7} A person familiar with Microsoft Word should be able to compose a text document there, and allow someone using ez to edit it, and return it without losing the Microsoft Word style formatting. \symbola{7} A person familiar with ez should be able to compose a text document there, and allow someone to use Microsoft Word to edit it and return it without losing the ez style formatting. } This is no mean feat, since RTF and ez support non-overlapping sets of formatting and do so in different architectures. The programs try to support everything possible in Word with ATK formatting, and to encapsulate as coded symbols any formatting that ATK does that would normally be lost in going to Microsoft Word. You are supposed to be able to edit in ez with formatting that Word would not normally allow, edit in Word and then recover the file in ez without losing the ez-specific format directives! As with their ancestor, \bold{scribetext}, the RTF translators are table driven. To enable Compiling of the RTF translators set \typewriter{RTF_ENV = 1} in your site.h file. \section{neos -- the Networked Educational Online System } Whenever I talk about NEOS, I always say that it is a "Classroom oriented file exchange system." The computer provides text editing functions, and file storage functions, and communications functions. However the abstractions of UNIX file, text editor, and network connection are computer concepts, not classroom concepts. In the classroom there are handouts and assignments. There is a process of exchanging papers among class members for comment, and another for submitting papers for grading and getting them back. The whole concept behind NEOS is to build computer abstractions that more closely represent processes and objects found in the classroom. NEOS builds a file repository that classifies files as handouts, exchangeables, and gradeables. The fxlib application programmer interface hides the implementation details of the file repository. Application programs submit to and retrieve from the repository through the fxlib API. The \bold{eos} \footnote{\ \begindata{fnote,539680264} \textdsversion{12} Originally the whole system was named EOS, not NEOS. But a large university computing facility subsequently adopted "Eos the goddess of dawn" as their totem in analagous fashion to the "Athena godess of wisedom" totem of MIT. We renamed the system NEOS (adding the word "Networked" to the acronym) and kept the program name the same for our users.\ \enddata{fnote,539680264} \view{fnotev,539680264,99,0,0}}application program allows a student to compose a paper in ez, submit it for grading, and retrieve an annotated copy of that paper online. The \bold{grade} application program allows a teacher or grader to browse a folio of submitted papers, edit the papers in ez, annotate, and return them to the student. The file repository understands access control lists in a way that closely mimics the model in the classroom itself. By default, anyone can submit a paper (in analogous fashion to the notion that anyone can walk into a classroom). Alternatively a class list can be created which limits submissions exclusively to members of the student list. Graders are known in advance and have special privileges. The security system enforced by Kerberos prevents students from modifying or even viewing the turned-in papers of other students. Further, it doesn't allow a student to modify a submission once made. A new version is added to the repository next to the old version. This same security system permits all graders to read, modify, and return submitted papers to students. You thus have the same situation in NEOS as you would have in a regular classroom if a person is both taking the class and grading papers in it. Although the turnin, annotate, return cycle is the most interesting case technically, the NEOS file repository understands handouts and exchangeables. Everyone in the class, both student and grader, can read and delete papers submitted for exchange. All students can read handouts which latter can be created only by members of the graders group. These applications have been in use for several years in certain sections of writing classes at MIT. In these "Electric" sections, students write in class on the computer and then comment on each other's writing without resorting to the time and paper consuming photocopying process. With a little imagination, one can conceive of other uses for eos and grade, such as cycling documents for composition/review/revision between worker and manager. Environments other than writing classes could use aspects of ez to permit the exchange of programs, tables, equations, for example. NEOS has an interesting place in the realm of computer mediated cooperative work. \subsection{Caveat: the NEOS build is a little tricky } To compile NEOS you must compile the contents of the \typewriter{fxlib} directory to get the server and the fxlib API. Then compile the neos subdirectory with \typewriter{NEOS_ENV = 1} set in your site.h file. Although NEOS is useful, we expect that people at other sites might have trouble getting it to work locally. NEOS consists of the client programs and a server. The client programs are straightforward. There is one set layered on ez, and another that is keyboard oriented and called from the UNIX shell. The server may present challenges to some sites: lack of Kerberos and a rough build process. Complete source for the server is included in the \typewriter{fxlib} subdirectory. However two essential libraries are not included: Kerberos authentication and encryption libraries \typewriter{libkrb.a} and \typewriter{libdes.a}. I didn't want to get tangled up in the red tape of exporting cryptography software, so you are required to get kerberos up yourself. Many sites already have Kerberos installed and have their user accounts set up to give every system user a Kerberos identity. The security features of NEOS which help to prevent plagarism and mutilation of student assignments is based on Kerberos. As an afterthought we did conditionalize the security on the \typewriter{KERBEROS} compile-time flag, but we've never attempted to compile the system that way. NEOS uses yet another unique set of Imake rules. The NEOS developers handed the system off to the people responsible for making the MIT campus wide software release. They have Imake rules which solve important problems for them. Rather than get into the business of maintaining multiple sets of Imake files, we include the Athena Imake configuration directory in fxlib/config. Build the top level makefile with \bold{xmkmf}, or by hand -- naming the config directory as the place to fetch \typewriter{Imake.tmpl} and \typewriter{Imake.rules}. The \typewriter{com_err} library is included in contrib/mit for your convenience but is also included in the Athena, and AFS distributions, so you could (and probably should) avoid compiling this one. (No point in duplicating what you already have.) I will not pretend that this is a nice neat integration and build procedure. It isn't. If others decide to compile up NEOS, and tell me what to change to make the build procedure smoother, I will try and make improvements for subsequent releases.\footnote{\ \begindata{fnote,539680520} \textdsversion{12} For a limited time, I will respond to email queries for help in setting up NEOS. Write to me at wdc@athena.mit.edu -- Bill Cattey -- Beleagured maintainer of ATK at MIT \enddata{fnote,539680520} \view{fnotev,539680520,100,13,0}} If building the server seems too difficult, follow the API spec and build your own server. A previous version of the API was layered on top of NFS and \bold{mount}. It should not be too difficult, for example, to build a repository conforming to the API that works using protected directories on a timesharing system rather than a Kerberos authenticated connection to a physically isolated service host out on the network. NEOS was written in 1989 as a project led by Professor Edward C. Barrett of the MIT Writing Program. Under the management of Bill Cattey of MIT Project Athena, students Rob French (MIT) wrote the server, Bruce Lewis (MIT) wrote the client library and command-line clients, and Nick Williams (Imperial College, London) wrote the X-window clients available in the andrew-neos distribution. There is further documentation of how MIT uses NEOS in the file \typewriter{doc/papers/Cattey.writing}, and how NEOS evolved in the file \typewriter{doc/papers/Cattey.turnin}.\footnote{\ \begindata{fnote,540033032} \textdsversion{12} these files are in the top of the Andrew Toolkit tree, not in the doc subdirectory of neos.\ \enddata{fnote,540033032} \view{fnotev,540033032,101,0,0}} I'm really proud of NEOS. It won an award for educational software: the Joe Wyatt Challenge. \section{Projects Under Development or Consideration } I've already mentioned that we are working on a subclass of the annot/icon object that would contain troff commands. That project is nearly done. I've described the direction we plan to take the GNU Emacs compatibility. We are also working on a directory of helps for constructing a thesis using ez. This will consist of a root file for use with datacat to contain the chapters, a title page, and various other useful little files together with documentation. We are working on two objects to contain bibliographic citations and a database of bibliographic information. We've been through several iterations of design and implementation and are slowly converging on something that our test users like. The goal is to be able to produce references in ez to a bibliography (perhaps multiple bibliographies) which will print correctly. The bibliographic database format would be compatible with existing Scribe and LAT\subscript{E}X bibliographic database formats. There is the possibility that some sort of prototyping system will enable people to specify how the bibliography is presented at print time. (After all, Scribe and LAT\subscript{E}X support such functionality.) Many students at MIT choose not to do their theses in ez because the eq equation editor is neither robust enough nor powerful enough to represent the equations in their theses. If my summer student is able to complete the projects already mentioned, he and I will be working on eq: 1. Survey what unexploited features exist in troff eqn. 2. Determine which of those features can be represented in the eq internal model of an equation. 3. Determine the relative priorities and implementation difficulties for making those representable features display in the eqview, and print. 4. Implement a pragmatic balance of high priority, low difficulty features. 5. Fix some of the stupid things eq does such as: \leftindent{"bar" hangs troff... An isolated zilch in a multi-line equation should be able to be deleted, decreasing the number of lines in the equation by one. } Two other projects which excite us but will probably not get support for anyone to implement are a continuous function plotting inset modeled after GNU plot, and a real WYSIWYG paste up system (like Page Maker) using the layout object and the ruler. \section{Where we would like to see ATK go in the future } For MIT there are two different markets for ATK. Future directions to propose for ATK can address one or both of these two markets: ez as a document preparation environment, and ATK as an implementation platform. \subsection{For the Document Preparation Market } Robustness and elimination of surprises is far and away the most important feature. There are many open bugs about how the troff postprocessor is hung, core dumped, or confused by user actions. There are still too many core dumps and strange redisplay and scroll related problems. These should be systematically eliminated. Perhaps ez should develop its own internal formatting system and migrate towards WYSIWYG. The integration with groff and ghostscript and the X PostScript extension represent excellent starting points. Another possibility would be to support T\subscript{E}X in addition to troff as a printing back end. I've already mentioned issues with eq and our interest in continuous function plotting. The lack of a decent drawing editor has also prevented use of ez in many document preparation tasks. The ATK newsletter mentioned a preference editor is in the works. This would be very good for our people. But preferences should be cleaned up internally and interfaced to the X resources system for those ports of ATK running under X. \subsection{For the ATK as Development Platform Market } For those who develop in ATK there are four obvious improvements: 1. Replace all those files that say "This is supposed to document such and such but there are no plans to write this doc yet." with real documentation. 2. Create a root file for use with datacat that understands and reasonably groups all the doc files. 3. Produce a new manual with a useful index. 4. Further tune ADEW and make it into a production level system. For clues on what's missing sit down in any of the new Motif toolkit application builders or NeXT Step. Effort to remove rough edges will be rewarded with increased market share. Another topic that would be difficult to get right but would immediately increase market acceptance of ATK would be interface to the Xt intrinsics. I think that a complete solution consists of two parts: An import wrapper that allows Xt based widgets to work under the Andrew Interaction Manager. An export wrapper that allows ATK based widgets to work within an Xt application. This could probably be done as a variation on the existing X interaction manager. \subsection{Other } These days there are many companies selling products that enable development of applications that run under X, Windows and Mac Toolbox from a common source tree. Many of these common toolkits contain trivial subsets of the Interaction Manager graphic and command protocol. If a port of IM were made to the Mac and Windows, it is conceivable that the ATK market could explode! Yes, there will always be applications better suited to single platforms, and developers who prefer the original toolkits. But some of the expensive garbage being hungrily devoured in the marketplace could easily be replaced by a cleaner solution based on ATK. Applications under ATK other than ez have been slow to be accepted. The static linking feature proposed in the mailing list will help. But more robust and cleaner applications are also called for. The messages mail reading/sending system is exciting! It has a beautiful user interface. Inside it is a pig! If the message server were torn out and replaced with something simpler that would interface to /usr/spool/mail, POP, and nntp, and if the statically linked image were 600K instead of 2 meg in size, I think you'd get a much larger user community. Thanks for listening. \enddata{text,539955892}