Lemur 3.0 release notes

• We have tested using gcc 3.2.2, 3.2.3, VC++ 6.0, and VC++ .NET .
  

• This version is NOT backwards compatible. For more information see the "Code Update" section below

• New Applications:
  • Cluster - This application performs the basic online clustering task. In conjunction with an incremental indexer (such as KeyfileIncIndex), it could be used for TDT topic detection task. It iterates over the documents in the index, assigning each document that is not in any cluster to a cluster. The document id, cluster id, and score are printed to the standard output.
  • OfflineCluster - This example application demonstrates the basic offline clustering task. Provides k-means and bisecting k-means partitional clustering. It will run each algorithm on the first 100 documents in the index (or all of them if less than 100) and print out the results.
  • PDictManager - Probabilistic Dictionary Maintenance
    Provides a command line interface for interacting with probabilistic dictionaries (PDict objects). Support creating a new dictionary, merging dictionaries, pruning one dictionary from another, normalizing probabilities, outputing the dictionary to a file, and interactively looking up entries.
  • PLSA - Probabilistic Latent Semantic Analysis
    This application will either perform PLSA on a collection, building three probability tables: P(z), P(d|z), and P(w|z) where z in Z are the latent variables (categories), d in D are the documents in the collection, and w in W are the terms in the vocabulary over the collection, or open those tables and read them into memory to illustrate their potential use.
  • XlingRetEval - An application for running cross-lingual retrieval experiments.

• Additions, Enhancements, and other changes:
  • New clustering support for online and offline clustering tasks. Provides k-means and bisecting k-means partitional clustering algorithms.
  • New support for cross-lingual language modeling and information retrieval.
  • New support for probabilistic latent semantic analysis.
  • New probabilistic dictionary class.
  • Better support for passage retrieval using any retrieval method.
  • New global Exception class for Lemur with macros for better exception handling. Supports nesting of messages, including file and line number information.
  • New static methods in RetMethodManager for running queries with one method call (RetMethodManager::runQuery). Retrieval method parameters are taken from the parameter stack or defaults used.
  • Improvements to the Lemur CGI, including showing DATE fields in results and showing document excerpts in results.
  • Improvements to the Lemur Retrieval GUI, including paging and cacheing of results, showing document excerpts in results, match highlighing using MatchInfo, saving program options between sessions, exception catching, and using native system look and feel.
  • New STL style external iterators for DocInfoList and TermInfoList. These iterators work for all lists currently implemented in the toolkit. No downcasting of lists returned from Index is required to use the iterator.
  • Promote position specific methods into general API (ie Index::termInfoListSeq, TermInfo::position, DocInfo::positions) so that downcasting of objects is not required to access position information. Objects that don't store position information, return NULL.
  • Broaden the use of different query objects with retrieval methods by creating new TermQuery class to serve as API for iterating over terms in a query. Changed TextQueryRetMethod and StructQueryRetMethod to use this in API instead of own query classes.
  • Updated StringQuery class can be used with any retrieval method expecting a TermQuery. StringQuery is also a TextHandler that can be used for parsing a query or catching query terms at the end of a stopper/stemmer/etc chain. *note* - constructor parameter changed from StringQuery(queryid, query) to StringQuery(query, queryid).
  • General improvements to the Lemur API by using string instead of char* and using const wherever appropriate. Other general improvements include adding of encapsulation methods to basic objects and removing many friend classes.
  • General deprecations of outdated or unnecessary objects. Please see "Code Update" section below for more information.

• Code Update:
  • This version has many API changes to comply with C++ const correctedness standards and changed most uses of char* to string objects. Don't be alarmed if you see a lot of errors when you try to compile your code against 3.0. Most of it is just simple fixes that involve adding const here and there to variables accepting return values from function calls and changing char* to string.
  • In general, other changes can be made to improve the performance of your programs, such as removing unnecessary downcasts and using new iterators, but they should still compile without those changes.
  • While you'll definitely need to recompile your code, you *should* be able to use new code with existing Inv(FP)Indexes, but we make no guarantees. KeyfileIncIndexes will definitely have to be re-indexed with the new toolkit.
  • Here is a list of deprecations. These are things that no longer exist in the toolkit:

    Deprecated:	Replaced with:
    class BasicDocInfo	class DocInfo
    class BasicTermInfo	class TermInfo
    class TokenTerm	class Term
    class StringToken	class Term
    class BasicTokenTerm	class Term
    class QueryToken	class Term
    class StructQuery	class TextQuery or class StringQuery
    application PushIndexer	application BuildInvertedIndex (identical)
    application BuildBasicIndex	the BasicIndex is being deprecated. use BuildKeyfileIncIndex to use KeyfileIncIndex or use BuildInvertedIndex for Inv(FP)Index

    
    
    
  • Here is a list of things that will be deprecated in the near future:

    Will Be Deprecated:	Replaced with:
    class InvDocInfo	class DocInfo
    class QueryDocument	class StringQuery
    class BasicIndex and its supporting classes	use Inv(FP)Index or KeyfileIncIndex
    sequential iteration methods in DocInfoList and TermInfoList (startIteration, hasMore, nextEntry)	STL style external iterators

• Bugs Fixed:
  1. Problem: Keyfile leaks memory on open or close
     Solution: Change use of auto_ptrs
     
  2. Problem: KeyfileIncIndex adds duplicate file and corrupts index counts
     Solution: Change to discard duplicate files (docids) with warning
     Problem: Keyfile index is ruined if a document with a long key is indexed
     Solution: Fixed to truncate long docids to maximum id length and discard terms longer than maximum length
     
  3. Problem: BuildDocMgr crashes if KeyfileDocMgr or ElemDocMgr already exists
     Solution: On build, check for existing DocMgr and call appropriate load method for incremental adds
     
  4. Problem: SimpleKLParameter values not set in RetEval, RelFBEval, TwoStageRetEval applications
     Solution: Call SimpleKLParameter::get in applications
     
  5. Problem: QueryClarity per term scores are incorrect
     Solution: Correctly scale term contributions so that they sum to the total clarity score.
     
  6. Problem: Memory leak in QueryModelEval
     Solution: Delete workSet after use.Also delete workSetStr and docPool.
     
  7. Problem: IncIndexer fails on windows. windows stdlib does not properly handle ios::app (mode "ab") for opening file streams.
     Solution: Replace uses with work-around
     
  8. Problem: Memory leak if Parser::setAcroList(string) is called more than once for same object.
     Solution: Delete old list before setting new one
     
  9. Problem: KStemmer seg faults if KstemmerDir path is invalid
     Solution: Modify to emit an error message and throw an exception if failure occurs
     
  10. Problem: Retrieval GUI crashes when using InvIndex or BasicIndex
      Solution: use Okapi as default method (no positions required) and add exception handling
      

 The Lemur Project

  Last modified: Wed Jun 30 14:21:51 EDT 2004