Pointing the Way: Active Collaborative Filtering David Maltz CMU Kate Ehrlich Lotus Presented at the 1995 ACM SIGCHI conference and published in the Proceedings. This document is made available under the ACM copyright rules and all readers must follow them. Abstract Collaborative filtering is based on the premise that people looking for information should be able to make use of what others have already found and evaluated. Current collaborative filtering systems provide tools for readers to filter documents based on which ones were read and liked by previous readers. This paper describes a different type of collaborative filtering system in which people who find interesting documents actively send "pointers" to those documents to their colleagues. A "pointer" contains a hypertext link to the source document as well as contextual information intended to help the recipient determine the potential interest and relevance of the document prior to accessing it. A preliminary version of our system has already proven easy t o use, with people using it to "bookmark" documents, send pointers to their colleagues and create "digests" that combine pointers with original text. Based on our experience we discuss the benefits of this form of filtering as well as its limitations. Keywords: Collaborative filtering, information retrieval, social filtering, hypertext, Internet, Lotus Notes. Introduction Connectivity, networking, and the National Information Infrastructure have become the buzzwords of the day. Underlying the excitement of these times is the promise that each of us will soon have unlimited access to a computer that will cheaply bring us information from sources around the world. We will be in direct contact with all the world's repositories of information -- no matter how small or large -- and in direct contact with the experts and people who create those repositories. Just like the upswell of creation and learning that followed the development of the printing press and the widespread access to information it created, we anticipate a new surge of knowledge that will enrich our lives. Unfortunately, our situation parallels that of the printing press in more ways than one. As the first libraries were built and books became available to larger groups of people, a new problem arose. Once buildings could be filled with more piles of books than any human could possibly read in a lifetime, how could people find books on the topics they wanted? The solution to that problem evolved into an entire field called Library Science. A solution to the problem of finding useful information on a global network promises to be no easier. Just to put some numbers on the size of problem, it is informative to look at several of the information systems available to users. The World Wide Web allows nearly anyone with a machine on the Internet to create hypertext multimedia documents and link them to other documents at other sites [1]. The Web started in 1990 and has grown to more than 4600 servers in October 1994, and perhaps over 10,000 servers in December 1994. [A] Together, these servers export a minimum of 1.05 million documents.[B] Usenet Net News is also growing exponentially. Estimates show that in May 1993 there were over 2.6 million users of Net News at some 87 thousand sites throughout the world. These users generated over 26 thousand new articles a day, amounting to 57 Mbytes of data [13]. Just over a year later, some estimates put the number of users at almost 10 million. The problem gets worse when one considers all the information that organizations are putting on-line in file folders, databases and Lotus Notes amongst other repositories. Not only is the amount of information available to many people far in excess of what can be retrieved through informal browsing methods, but only a very small fraction of the available information will be accurate, up to date and relevant. Building on the notion of collaborative filtering which originated with the Tapestry project at Xerox PARC, this paper describes a system that supports the informal practice of sharing pointers to interesting documents and sources of information with colleagues. We begin by reviewing existing information filtering systems. Information Filtering If the problem is that people are swamped by too much information [12, 16], the solution seems to lie in developing better tools to filter the information so that only interesting, relevant information gets through to the user. Many present filtering systems are based on building a user profile. These systems attempt to extract patterns from the observed behavior of the user to predict which items would be selected or rejected. Examples of this form of filtering include LyricTime [10] which compiles a profile of musical preferences, an agent that filters NetNews [15] and INFOSCOPE [4] an interesting hybrid news reader that frames the filtering problem as one of restructuring the information space on a user by user basis to place all the articles relevant to a user in a few known and accessible locations. However, these systems all suffer from a "cold-start'' problem. New users start off with nothing in their profile and must train a profile from scratch. Even with a starter profile, there is still a training period before the profile accurately reflects the user's preferences. During the training period the system can't effectively filter for the user. A better system would allow new users some type of access to the experiences of current users to create a better initial profile. The second drawback is that the user's searches can become circumscribed by the profile. The profile only selects articles similar to the ones the user has already read, and new areas that might be of interest can be missed completely. If the user tries to explore new content areas a new profile must be created that is customized to that subject matter and the user again faces the "cold start" problem. Collaborative Filtering Most present filtering systems fail to capitalize on a key resource made available by on-line information systems, namely the knowledge and wisdom accumulated as different people find and access documents and form opinions of them. By giving users access to others' prior experience with an information source, we can create a collaborative information filter. Collaborative filtering systems work by including people in the filtering system, and we can expect people to be better at evaluating documents than a computed function. Current automatic filtering systems attempt to find articles of interest to their user, often using some scoring function to evaluate features of the documents and returning the documents with the highest scores. People can effortlessly evaluate features of a document that are important to other people, but would be difficult to detect automatically. Examples of such features are the writing style and "readability'' of a document, or the clarity and forcefulness of an argument the document contains. Imagine the difficulty an automatic filtering system would have figuring out which of two cake recipes is "easier to follow.'' Another motivation for collaborative filtering comes from comparing the rich environment of real objects to the much poorer one in which computer users operate. When a user reads a computer file he usually has no way of telling whether he is the first person to ever read it, or if he is looking at the most commonly used reference on the system. Collaborative filtering works in part by associating with computer documents the history of their use. For instance, Hill et al make the observation [7] that the objects we use in everyday life accumulate wear and tear as a normal part of their use: pages in books become wrinkled, bindings creased, and margins smudged with fingerprints. The objects with more wear are the more commonly used ones, and further the wear acts as an index to relevant information inside the object. An example is the way reference books open to commonly used pages when dropped on a desk. Giving searchers access to this usage history lets them take advantage of the type of subtle hints that we commonly use when making read/don't read decisions in the real world. Tapestry The concept of collaborative filtering originated with the Information Tapestry project at Xerox PARC [5]. Among its other features, Tapestry was the first system to support collaborative filtering in that it allows its users to annotate the documents they read. Other Tapestry users can then retrieve documents to read based not only on the content of the documents themselves, but also on what other users have said about them. Tapestry provides free text annotations as well as explicit "likeit'' and "hateit'' annotations so users can pass judgments on the value of the documents which they read. In its current incarnation, Tapestry suffers from two distinct problems. The first problem is the size of its user base. Because Tapestry is based on a commercial database system it can not be given away freely. Further, Tapestry was not designed for use by large numbers of people at distributed sites. Both these factors combine to limit the pool of potential Tapestry users to researchers at Xerox PARC. Based on anecdotal evidence, this pool does not seem large enough to support a critical mass of users. The vast majority of documents go unannotated, so there is little collaborative information to use when filtering. The second problem with Tapestry is the means by which users enter filters into Tapestry. One common interface to Tapestry requires users to specify requests for information in the form of queries in an SQL-like language. Writing such a query requires the user to have a firm sense of what types of articles he wants to read, which is a hindrance to exploration of new areas and makes it hard to browse the available information for serendipitous hits. Collaborative Filtering of USENET Net news A further collaborative filtering system for Usenet Net News was created to scale up to a critical mass of users by branching out to more sites, and providing those users with a simpler method for accessing articles [11]. This system modified two common Net News readers to provide collaborative filtering functions: voting and display of articles based on their "popularity''. Aside from these extra functions, the news readers appear and handle as they did before. Users of the modified news readers were encouraged to cast votes for or against the articles they read. These votes are used by the system to create a net-wide collective opinion on the usefulness of each article. The news reader clients can then use this information to help future readers of the newsgroup find articles of interest. Users are able to associate their names with their votes or not as they chose. A positive result of the work on this system was finding that people really will vote. Although no reward or incentive of any kind was offered to the users of one of the modified news readers, 24 of the approximately 40 users of the newsreader voted for articles during the course of one three month study period. However, this number was insufficient to reach critical mass. That is, because of the large number of different documents, this system was dependent on lots of people reading and voting on the same documents. Users reported that when they didn't see any votes in the groups they read, they thought the system must be broken or not working so they didn't bother to vote further because their votes would be lost and wasted. If they knew something useful was happening to their votes, they would have kept voting. GroupLens A similar system for NetNews, GroupLens, combines collaboration with user-profiles [14]. Communities of users rank the articles they read on a numerical scale. The system then finds correlations between the ratings different users have given the articles. As viewed by a user called Jane, the goal of the system is to identify a peer group of users whose interests are similar to Jane's, and then to use their opinions of new articles to predict whether Jane will like the articles. Like other filters based on a user-profile, GroupLens suffers from the cold-start problem. One problem with many of these collaborative filtering systems is that they require a critical mass of users to vote or leave their mark for any aggregate score to be meaningful. Until systems such as the NetNews collaborative filter are well in use, there are uncertain rewards for any user who participates. The lack of a clear reward system can be a major barrier in the acceptance of a groupware application [6]. Some systems also suffer from usability problems due to excessive overhead in either registering a vote, accessing the result of other people's votes, or creating a profile. "Active" collaborative filtering These three examples of collaborative filtering are well suited to situations where there are a lot of documents in a single database such as NetNews. In these cases, there is a benefit to aggregating the votes of the many readers who access the database. We call this "in-place" or "passive" filtering because there is no direct connection between a person casting a vote and the readers who come later and filter documents based on these aggregated votes. Another approach to collaborative filtering, and one that we have adopted here, builds on the common practice where people tell their friends or colleagues of interesting documents. We call this "active" collaborative filtering because there is an intent on the part of the person who finds and evaluates a document to share that knowledge with particular people. For example, as part of the World Wide Web system users collect "hotlists'' which are effectively lists of hypertext links to the interesting World Wide Web pages that they have found. Several simple systems have been developed to help users format and distribute these hotlists to others, thereby spreading information about good documents on the Web. Another example in the World Wide Web context is Simon [9]. Users of this system create "subject spaces'' which are effectively lists of hypertext links to the interesting World Wide Web pages that have been found, and comments on those documents. Individual people can use subject spaces to keep track of their own explorations, but they can also send their subject spaces to a group Simon server. However there is no provision in this system for sending hypertext links to particular individuals. We see "active" collaborative filtering being useful in cases where users are not familiar with the whole range of databases that are available. Thus, we see it being applicable for people using the World Wide Web, on-line information services, Lotus Notes databases and other similar large collections of diverse information. In contrast with "passive" collaborative filtering where the system works better the higher the convergence of votes on the same set of documents, "active" collaborative filtering works best when there is divergence among users in which documents they select since that will identify a larger number of documents. The benefit of an active filtering system is also motivated in part by a recent study of a customer support group [2]. Ehrlich and Cash reported that people in the group had frequent informal ad hoc meetings to solve customer problems collaboratively. In these and in group meetings, support analysts would routinely tell each other about useful sources of information. Moreover, there was one person in the group who was especially skilled at finding relevant information and applying it to the problem at hand. He was remarkable for his breadth of knowledge of the domain as well as of useful sources of information, general problem-solving, and, communication skills. By virtue of these skills he assumed the role (though not the title) of an information mediator. That is, he was available as a resource to other analysts in the group to help solve especially troublesome problems. Part of his job was authoring documents for a "knowledge" database the group maintained for storing troubleshooting scripts and other information not easily found in existing resources. That is, because of his recognized authority in the area, it was expected that the information he published would be accurate. This notion of ensuring the quality of information is well known to librarians for whom source validation is an important part of their job [3]. Interestingly, the need for validation is especially important for documents published and distributed on-line. Most printed material has undergone a review or editorial process before being published. But a lot of material gets distributed on-line in newsgroups, discussion databases and the like without benefit of any review or selection process. A key motivation for developing our active collaborative filtering system was to develop tools that would support the information foraging and source validation skills of users like librarians or "information mediators". Description of the System As we noted earlier, the practice of sharing references to documents with colleagues is a common one. But many of the informal methods of sharing the information have drawbacks or limitations. The system described in this paper was developed around three key ideas that we felt were missing from existing informal systems. 1. Package contextual information with hypertext links. Existing methods for sharing references to on-line documents are often limited to just the hypertext link, perhaps with a few comments. Yet additional contextual information about the name or location of the source, the date of the document as well as kno wledge of the sender's selection biases can be used to judge the relevance of a document prior to reading it. 2. Ease of use. Informal systems are often awkward to use. For instance the ability to add annotations to the hypertext link may not be readily available. Not only should these features be easily accessible but the system in general should return value to the senders and recipients early on to encourage usage. It is through continued and broad adoption, that an active filtering system demonstrates its usefulness. 3. Flexible. The main drawback of many of the informal systems is that they work well for a particular situation but are not flexible enough for general use. We intend to explore, for instance, different methods of distributing the package of hypertext link, comments and context to probe the potential extendibility of our system. Pointers The basic concept in our system is the "pointer". Modeled after what people do informally, a pointer contains a) a hypertext link to the document of interest; b) contextual information - title and date of document, name of source database, name of sender, and c) optional comments by the sender. A typical pointer is shown in Figure 1. Figure 1: A sample pointer as seen by a recipient We implemented our active collaborative filter inside the Lotus Notes environment. Notes is a commercial product that provides a client-server platform for developing and deploying groupware applications. At the user level, Notes consists of documents which are stored in databases on server machines. These documents can be thought of as raw records of field-data pairs. Databases are typically organized around a topic, and the documents they contain can either be created by users inside of Notes, or gatewayed into Notes from external information sources such as Net News, World Wide Web, or clipping services. In addition to documents, each database has a set of forms and views. Database forms can be thought of as document templates containing fields that are filled in during composition time. For instance, the form used to generate the pointer in Figure 1 has a field for each item. Each database has one or more views which are ways of grouping and displaying the list of documents in the database. For instance, documents could be listed in order of document author, document date or category name. The latter being one or more keywords defined by the user to describe a document. Typical applications in Notes are developed by creating forms, views and databases to support a new type of information, and then adding macros which specify, automate, or regulate the actions that can performed on the information. Notes allows these macros to be embedded directly into the document as buttons whose macros operate on the document's data. Notes' data management infrastructure also provides the equivalent of a hypertext link called a doclink. Doclinks present themselves on the screen as small page icons; clicking on one of these icons opens the referenced document as a new window on the workspace. Pointers are implemented in Notes as a special form. These forms are accessed from a button we added to the Lotus Notes SmartIcon (TM) bar. Pressing the button while a document is selected will open one of these forms and fill in the document title, creation date and database name of the currently selected document using heuristic rules. The user can add comments of any length. Since all documents in Lotus Notes are semi-structured by the forms used to create them, the information extraction heuristics are, in general, very effective. In addition to information about the pointed to document, the pointer itself includes buttons which help the sender to distribute the pointer to selected people and in a variety of ways --- this distribution process is described below. In order to give pointers some functionality which could not be easily implemented from inside of Notes, our collaborative filtering system also involves a small server process which runs outside of Notes. The buttons on a pointer form communicate with this process which then implements the pointer distribution methods, making it easy to experiment with different distribution methods by altering the server. Distribution of Pointers Once a sender has created the pointer, he/she can save it in a private or public database, email it to one or more people or distribution lists, or save it in a special form we call an "Information Digest". These distribution methods are depicted in Figure 2. Figure 2: Different ways of distributing a pointer. Private or group database. At the simplest level, users can "bookmark" favorite documents by saving a pointer in a private database. We think of these private databases as a scrapbook for saving pointers to interesting documents especially those we might not have time to read immediately. Users can add keywords to their pointers to help organize their scrapbook. The user can view their list of pointers in various ways including by date of source document, name of database or keyword category. Although not yet implemented, the system allows for pointers to be saved in a public database where anyone with the appropriate access rights can add pointers or read them. Email and distribution lists. Pointers can be sent to other people via the email system. Figure 1 shows how a pointer might look when it is opened. Whether from a database or email (which is only another database in Notes), the user opens the referenced document by clicking on the doclink icon. It should be noted that because a pointer is a regular Notes document, anyone running Notes can receive a pointer and follow the hypertext link to the referenced document; they don't need to be running the collaborative filter. Moreover pointers can be forwarded just like any other mail message. Pointers can also be sent to a distribution list of people who have previously registered themselves as interested in receiving pointers from the evaluating user, similar to a subscription service. This means of distribution gives our system a method of filtering akin to Tapestry's notion of filtering for messages based on who has annotated them. In Tapestry, users can create filters of the form "show me the messages that Joe Blow has annotated." In our system, users can register themselves as being interested in receiving pointers that Joe Blow decides to distribute. Joe himself does not have to worry about who has registered their interest as the system will automatically lookup the registered users and mail out copies of the poi nter when Joe presses the Distribute button. We considered calling the Distribute function automatically when any pointer is created, but decided that failed to give the user enough control over who sees their pointers, an especially important concern since we want the system to be useful to people for keeping pointers for their own use. Information Digests. We have also begun exploring an advanced use in which multiple pointers are saved directly into a pre-designed document containing a combination of original text and pointers. Newsletters, World Wide Web home pages, company profiles, summary reports or even a table of contents that references documents from many different databases, are all examples of this kind of document. A portion of a typical digest is depicted in Figure 3. It consists of a banner title, a subject and several sections. As we found in [2], there are people who excel at pulling together bits of information from many places. On Usenet Net News, these people appear as the ones who pull together FAQ's (Frequently Asked Questions) for newsgroups. One form of an information digest might be an on-line newspaper or magazine written by people skilled in selecting, editing, annotating and placing stories together. Information digests support a style of working in which the writer collects references to background literature continuously over an extended period and at a separate time from when the original text is written. Although not implemented, there is no reason why information digests could not have multiple authors. .Figure 3: A sample Information Digest containing pointers formatted by our system and narration text added by the author. The pointers are indicated by arrows while the added text is in Helvetica font. {AXE THIS NEXT PARAGRAPH OR FOLD IT INTO ABOVE PARAGRAPH?? -DAM} Users can easily add a pointer to any section of a defined digest. The system automatically extracts the information from the pointer and formats it into the digest. Our system encourages the behavior students are taught; "fill out an index card on each reference as you find it and then later organize the cards into sections of a report". By simply collecting numerous pointers together we have found the digests to be quite useful. Users can edit a digest like any other document to rearrange the pointers and add in narrative or supporting text Passive filtering. For completeness, our prototype also contains methods to support passive collaborative filtering in addition to active filtering. If a user comes across an interesting document but doesn't know of anyone in particular the document should be brought to the attention of, she can invoke the passive filtering system on the document. The system will then annotate the document in-place so that any future readers of the database can use the annotation to help filter for useful documents. The passive filtering system works by marking the annotated document as having been read by the annotator, and creates a response note to the document which contains any comments the annotator has on the document. Our initial goal was to allow users to mark up the annotated document directly, but this turned out not to be feasible in the current version of Notes. An active collaborative filtering system similar to ours could be created in the World Wide Web environment quite simply by better integrating Web browsers with email readers. Hypertext links would come almost for free, while information digests could be represented as new Web pages. Private and group databases of pointers might be hard to implement, as no Web clients we know of currently support organizing multiple views of data without external database backends. Automatically extracting contextual information from Web documents may also pose difficulties. Some contextual information would be easy to obtain, such as the document title. Providing contextual information about the area of Web containing the document would be much harder, since by nature the Web has less structure than Notes databases. Based only on personal experience, it seems like providing both a link to the interesting page and a link to the parent page on which the interesting link was found goes aways toward providing recepients with some context on the region of the Web the page was found. Uses of Pointers A working version of the system has been distributed to about 20 people inside and outside our own workgroup. The system was distributed to people in an email message that contained a brief user's guide and the files that made up the system. People use it as a personal productivity tool - saving pointers to interesting articles, as well as for sending pointers to others in mail. We even have an example of one user creating an Information Digest. When asked to summarize a database of action items for a supervisor, this user created an information digest with two sections: one for hot action items and one for cool items. She then created pointers to the relevant documents and added the pointers into the digest while classifying the documents by placing some pointers into each section of the digest. Finally she mailed the digest to her supervisor as the summary. Although we have not conducted a comprehensive user study, we did learn a few things relative to the goals we outlined earlier: Packaging contextual information with hypertext links. We argued earlier that merely distributing a hypertext link is not sufficient, even if it includes comments by the sender. Recipients need contextual information to evaluate the usefulness of the document. The additional contextual information supplied as part of the pointer has been used as the basis for organizing the list of pointers in a personal database. One user reports using that feature to see which database seems to yield the best or highest number of pointers. Recipients who get pointers in email are also appreciative that they can see the database name along with the hypertext link. There have been several instances when recipients learned of new Notes databases they had not seen previously. Easy to use. The system was designed to be easily used both by the person creating the pointer as well as by the recipients. Saving a pointer to a personal database is so easy, some users have already amassed quite a large collection. A pointer can be saved with only 3 keystrokes: selecting the pointer SmartIcon, pressing Esc to save to the database and clicking "Yes" to the query about saving the document. After the system was released, we also had very few requests for help from people using it. Although not stated as a goal, we wanted to avoid the problems plaguing some of the other systems we described where there is a considerable period of either training or critical mass that must be overcome before the system is functioning as intended. An active collaborative system such as ours starts being useful when the first person creates a pointer. Such systems also have the advantage that recipients can benefit from pointers found by others without themselves having to run the system. Flexible. One of our goals was to build a system that is flexible and extensible enough to be used in a variety of ways. A key area for extending our system is the information digest. While we have only just begun to explore the significance of this method of authoring and distributing information we believe it begins to offer new opportunities for sharing information. For instance, one of our users puts together a monthly on-line newsletter. He sees the information digest as a way of letting him do his job much faster and hence freeing up time for other important work. Active vs Passive Filtering Although we have enjoyed some success with this system, it has been used primarily within a small workgroup where people are quite familiar with their colleagues current interests. We also have only used the system for a short period of time and so don't have any data on scaling problems. For instanc, if lots of people were to send a pointer to the same document to any single individual, that person might get annoyed at the redundancy. On the other hand, a future enhancement could be a tool which automatically checked for duplicates (the linked document would be the same) and concatenated the comments from each message into a single, longer pointer. In general there are several major drawbacks to an active filtering system. One is that control over the document selection resides with the sender not with the recipient and puts the recipient in a passive role. This means, for instance, that the recipient cannot use an active filtering system to find filtered/reviewed information on a particular subject unless a sender happens to have sent out a pointer to such a document. On the other hand, active filtering systems have the advantage of simplicity and immediacy - no cold-start or critical mass problems to overcome. They also address a need among new users to learn more about the available information space as a whole. By giving everyone in a workgroup the opportunity to save and share pointers, the burden of finding new and interesting relevant documents becomes a shared exercise. There is intrinsic reward in participating for those people who enjoy browsing around looking for information they can share with others. Discussion In this paper we have taken a new look at the old problem of how colleagues share information about the location and existence of interesting documents. Motivated by a study of how such information sharing typically occurs in workgroups, we implemented a system which we believe supports that sharing. The key concept behind our system is to give the people who find information an active role in distributing that information to others. The key artifacts we provide to support that task are pointers which automatically gather contextual information useful to the pointers' recipients, and a distribution system for easily creating pointers and moving them to intended recipients. Having the users who find the information also responsible for sending it to colleagues, contradicts a common theory on improving information systems. Namely that information finders should be freed from the task of addressing mail and coming up with recipients [16]. Yet, at least one of our studies showed that often users really do have recipients in mind for the information they discover, and we believe our active collaborative filtering system serves users well by allowing them to easily act on this knowledge. Further, since the user who discovers a piece of information is mostly likely the one who knows how it should be fit in with other information, it makes sense to us that the information finder should have ways of easily writing that meta-level kn owledge down. By providing support for an existing means of information sharing, we believe we leverage the best of both worlds: high quality information filtering for recipients plus easy, immediate sharing tools for the senders. Acknowledgments The first sections of this paper borrow from work done at the Xerox Palo Alto Research Center as part of Maltz's SM thesis for the Massachusetts Institute of Technology. We extend our thanks to David Goldberg of PARC and Karen Sollins of MIT who supervised the thesis work and provided valuable comments on this paper, and to Irene Greif and John Patterson for their insightful reading of a previous draft of the paper. M.J.Johnson@qmw.ac.uk References Danzig, P., Obraczka, K., and Li, S-H. Internet Resource Discovery Services, IEEE Computer, September 1993, pp. 8-22. Ehrlich, K. and Cash, D. Turning Information into Knowledge: Information Finding as a Collaborative Activity. In Proceedings Digital Libraries '94 (College Station, 1994), 119-125. Ehrlich, K. and Cash, D. I am an Information Waitress: Bringing Order to the New Digital Libraries. 1994. Submitted for publication. Fischer, G. and Stevens, C. Information Access in Complex, Poorly Structured Information Spaces. In Proc. CHI'91 Human Factors in Computing Systems (New Orleans,1991), Addison-Wesley, pp. 63-70. Goldberg, D., Oki, B., Nichols, D., Terry, D.B. Using Collaborative Filtering to Weave an Information Tapestry. Communications of the ACM, December 1992, Vol 35, No12, pp. 61-70. Grudin, J. Why CSCW Applications Fail: problems in the Design and Evaluation of Organizational Interfaces. Hill, W., Hollan, J., Wrobleski, D. and McCandless, T. Edit Wear and Read Wear. In Proc. CHI'91 Human Factors in Computing Systems (Monterey, 1992), Addison-Wesley, pp. 3-9. Internet Society Press Release 4 August 1994 (rev.) Johnson, M. Simon Homepage: Welcome to SIMON,'' University of London, available via World Wide Web at http://web.elec.qmw.ac.uk/. Loeb, S. Architecting Personalized Delivery of Multimedia Information. Communications of the ACM, December 1992, Vol 35, No 12, pp. 39-48. Maltz, D.A. Distributing Information for Collaborative Filtering on Usenet Net News. MIT Department of EECS MS Thesis (May, 1994). Also available as Tech Report MIT/LCS/TR-603. Palme, J. You have 134 unread mail! Do you want to read them now? Proceedings IFIP WG 6.5 Working Conference on Computer-Based Document Services, Nottingham England May 1984, pp 175-184. Reid, B. Usenet Readership Summary Report For May 93,'' Usenet news.lists, 2 June 1993. Resnick, P., Neophytos, I., Mitesh, S. Bergstrom, P. and Riedl, J. GroupLens: An Open Architecture for Collaborative Filtering of Netnews. In Proc. of CSCW '94: Conference on Computer Supported Cooperative Work (Chapel Hill, 1994), Addison-Wesley. Sheth, B., and Maes, P. Evolving Agents for Personalized Information Filtering. Proceedings of the Ninth IEEE Conference on Artificial Intelligence for Applications, 1993. Terry, D.B. 7 Steps to a Better Mail System. Proceedings IFIP International Symposium on Message Handling System and Application Layer Communication Protocols, October 1990.