Installing and Running Lemur (Version 3.0)

Contents

1. Installation on Unix


2. Installation on Windows (NT and XP)


3. Running Applications


4. Testing the Toolkit on Sample Data


5. Using the API to Write Your Own Application


6. Modifying the Toolkit Libraries

1. Installation on Unix

After downloading the Unix Lemur package, follow the following steps to install it:

1. Unpack the source



On the command line, type in the following commands to unpack the package. This should create a directory named lemur-3.0.

> gunzip lemur-3.0.tar.gz 
> tar -xvf lemur-3.0.tar

2. Configure the makefiles

Go to directory lemur-3.0 and run the configuration script configure. This will generate a file named "MakeDefns", which has some customized definitions to be used in makefiles.
configure accepts the following arguments:


--enable-distrib compiles and installs the distributed retrieval components. Default is disabled.
--enable-summarization compiles and installs the summarization components. Default is disabled.
--enable-cluster compiles and installs the clustering components. Default is disabled.
--prefix= Specifies the directory for the installed toolkit. Default is /usr/local.




For example, to configure Lemur with the default libraries:

lemur-3.0>./configure 


Or to configure Lemur with some modules:

lemur-3.0>./configure --enable-distrib --enable-summarization

3. Compile Lemur

With directory lemur-3.0 as the current working directory, type in "make". This will compile the whole Lemur toolkit and link all the Lemur applications.

lemur-3.0> make 

4. Install Lemur library

After compiling Lemur, type in "make install". This will install the Lemur library and include files according to the directory specified by the prefix option of the configure script.

For example, with C-shell,

lemur-3.0> ./configure --prefix /usr0/mydir-for-lemur
lemur-3.0> make install

will create /usr0/mydir-for-lemur/lib/liblemur.a and a bunch of ".hpp" (C++ header files) and ".h" (C header files) in /usr0/mydir-for-lemur/include/. The application executables will be all in /usr0/mydir-for-lemur/bin.

For users who are only interested in using Lemur as a library and application suite, the original source tree (i.e., the lemur-3.0 directory) can be removed after this step.


5. Problems with installation
   
   We have dropped the support for any version of gcc older than 3.0. Solutions to some problems with installing Lemur have been posted on the Lemur Forum.

2. Installation on Windows (NT and XP)

• Extracting toolkit files


After downloading the toolkit, uncompress it. Even though it's a .tar.gz, winzip can do this if you simply double-click on the file. Choose a directory to extract the files into. The extraction process will create the lemur directory and all folder structures for you.


• Building the libraries


There are a number of .mak files in the lemur directory that are NMAKE files, one for each library. These can be built using the command line or using Visual C++ (version 6.0). The six files are lemur_utility.mak, lemur_index.mak, lemur_langmod.mak, lemur_retrieval.mak, lemur_distrib.mak, and lemur_summarization.mak. Each one builds a .lib with the same name in the same directory.


• Using Visual C++




In Visual C++, open the .mak file for the library that you want to build. It will complain about not having a project file and make one for you. Say yes, and save it (anywhere). It will create a project where the only file is the makefile. When you build, it might claim to build a .exe, but it will still build the .lib, and regardless of where you saved the project, it should build the .lib in the lemur directory. For more information, look in VC++ help, indexed under "makefile: porting NMAKE projects to the development environment."

If you want to create a real VC++ project for a library, you can create a new "static library" project and add in the source files from that library into the project. Each library contains all the source files from the directory with the corresponding name. For example, lemur_utility builds all files from lemur\utility and lemur_index builds all files from lemur\index.



• Using command line nmake


On the command line in the lemur directory (you will need to have nmake installed and in your path), using

nmake /f "lemur_utility.mak"

to build the Release version of the lemur_utility.lib, Or type

nmake /f "lemur_utility.mak" CFG="lemur_utility - Win32 Debug"

for the Debug version.


• Building an application


To build an existing application or to write your own, compile the main source file and link in the library files.

In Visual C++, create a new project (empty win32 console application) and add the source file for the application into the project. Link in the library files by doing this:

1. Go to "Settings" under the "Project" menu.
2. In the "Settings for" drop down list in the top left, select "All Configurations".
3. Click on the "Link" tab on the right.
4. List the libraries that your application needs in the "Object/library modules" field, ie lemur_utility.lib.
5. Choose "Input" from the "Category" drop down menu.
6. Make sure you have the lemur path in the "Additional library path" field.
7. Click on the "C/C++" tab.
8. Choose "C++ Language" from the "Category" drop down menu.
9. Check the "Enable Run-Time Type Information (RTTI)" check box.
10. Choose "Preprocessor" from the "Category" drop down menu.
11. In "Additional include directories" field, list the include directories for the Lemur libraries that you are using (comma separate multiple directories). For example: lemur-2.1\utility\include,lemur-2.1\index\include

3. Running Applications

The executables for Lemur applications are generated in the directory app/obj; they will be copied to LEMUR_INSTALL_PATH/bin after running "make install'.

The usage for different applications may vary, but most applications tend to have the following general usage.

Create a parameter file with value definitions for all the input variables of an application. Terminate each line with a semicolon. Note that the use of the semicolon is mandatory.For example,

dataFiles = /usr3/web/sourcelist;
index = /usr3/web/myindex;
memory = 128000000;
docFormat = web;
position = 1;

In general, all the file paths must be absolute paths in accordance to your operating system. Lemur does not have the capability of searching for files along different paths.

Run the application program with the parameter as the only argument, or the first argument, if the application can take other parameters from the command line. Most applications only recognize parameters defined in the parameter file, but there are some exceptions.

For example, if the parameter file above is named buildparam in the directory /usr3/web, then just do:

/usr3/web> BuildInvertedIndex buildparam

For new versions of gcc, you would need a shared library to run an application, and you will need to set the environment variable "LD_LIBRARY_PATH" to the path to the corresponding shared library. See gcc documentation for more details about this.

Most applications will display a usage or a list of required input variables, if you run it with the "--help" option. For details about how to use each of the applications in the Lemur toolkit, see the Lemur Applications User's Guide .

4. Testing the Toolkit on Sample Data

The Lemur Toolkit comes with a sample data directory which includes a small public information retrieval testing collection (i.e., the CACM collection available from the Cornell ftp site ftp://ftp.cs.cornell.edu/pub/smart/). This sample data is to let you easily try the toolkit and will help you to understand the capabilities of the toolkit as well as how to use them.

The directory has three shell scripts test_basic_index.sh, test_pos_index.sh, and clean.sh. test_basic_index.sh is a script that uses the basic index and demonstrates most of the functionality of Lemur, i.e., from formatting a database, building an index, to running various kinds of retrieval experiments. test_pos_index.sh does the same thing, but using the position index. (See the API documentation for more information about these two indices.) clean.sh cleans up any files generated by any of the two testing scripts. test_basic_index.sh and test_pos_index.sh are both self-documented, so if you look at the scripts you will be able to understand what they do.

Your output should not be too different from the output contained in the sample output files listed here. Roundoff error should only lead to minor deviations from these results.

Basically, both test_basic_index.sh and test_pos_index.sh would start from a source database file and a query file with some simple SGML format, and build an index of the database and a support file that is necessary to make some retrieval algorithms fast, and then, they will run different retrieval experiments with different parameter files. The retrieval results are evaluated with a perl script ( ireval.pl ) in the app/src directory. A precision recall summary file is generated for each experiment.

You can try to change some of the settings in the parameter files and see how it will affect the retrieval performance.

Windows users might be able to run these scripts under cygwin. Even if they can not run the scripts directly, they should still be able to repeat the commands in these shell scripts manually or in some other automatic way.

5. Using the Lemur API to Write Your Own Application

To use the Lemur API on Unix, you will need both the Lemur library file and all the header files. The installation script of Lemur generally puts the library file in LEMUR_INSTALL_PATH/lib/liblemur.a and all the header files in LEMUR_INSTALL_PATH/include/. Header files in C have the extension of .h, while a C++ header file has an extension of .hpp. You will use the Lemur library exactly in the same way as you would use any other C++ library. This means you generally do the following:

• In your C++ code, include the relevant Lemur header files.
• When compiling your code, use -I<LEMUR_INSTALL_PATH>/include as an option so that the compiler can find the included files. (Of course, you need to replace <LEMUR_INSTALL_PATH> with the actual directory where you install Lemur.
• When linking your code, use -L<LEMUR_INSTALL_PATH>/lib as an option so that the linker can find the Lemur library. Also, you need to specify -llemur as a linking option to indicate that you want the Lemur library to be linked with your code. You may need to be careful about the order of the libraries you specified. The order reflects the assumed dependency among the libraries.

An application level Makefile that you can use for your own applications has been included. To use it:

1. Copy Makefile.app from the top level lemur directory to the directory with your application's source code. Edit the file and fill in values for the following:
   OBJS -- list of each of the object files needed to build your application.
   PROG -- name for your application.

2. Use make -f Makefile.app to build your application.

To use the Lemur API on Windows, see Installation on Windows (NT and XP)

6. Modifying the Toolkit Libraries

1. The directory structure and makefiles

   The toolkit has two types of directories: (1) module directories ("code" directories) (e.g., index, retrieval , and app) (2) others (e.g., data). The code exists in all the module directories. Each module directory has four subdirectories: include, src, depend, and obj. "include" has all the header files, while "src" has all the implementation files. "depend" is to store a depend file for each source file. "obj" is to hold the compiled object code. obj and depend are created automatically by the makefiles. The data directory stores a sample testing collection and testing scripts. There are two types of modules: library module and application module. A library module will be compiled and linked as a module library, whereas an application module has a bunch of application programs. Each of them has a main() function. Thus, all the applications must be linked individually. For example, "utility", "index", "langmod", and "retrieval" are library modules; only "app" is an application directory.

2. To modify an existing file or add a file to an existing library module:
   1. Make the changes
   2. Go to the Lemur root directory
   3. Type in "make" to update the whole toolkit.

3. To add a new (library) module to the toolkit:
   1. Create the module subdirectory in the lemur root directory.
   2. Put all include files in a subdirectory named "include" under the new module directory
   3. Put all implementation files in a subdirectory named "src" under the new module directory.
   4. Add the module directory name to the Makefile variable LIBDIRS Note: If you rerun configure, you will have to make this change again. Advanced users should edit configure.ac and add an AC_ARG_ENABLE for the new module (see the distrib entry in configure.ac) and then use autoconf to generate a new configure script.
   5. Copy a Makefile from an existing module directory (e.g, index/src/Makefile) to /src, and change the variable MODULE to the name of the new module

---

The Lemur Project

Last modified: Thu Sep 9 20:41:13 EDT 2004