Package mekano

source code

Mekano

Provides low-level building blocks for information retrieval and machine learning, with a special focus on text processing.

Features

• Representing text documents as sparse vectors
• Representing a collection of documents as a dataset, which can be subsetted for cross-validation etc.
• Evaluation using various metrics
• Reading various common input formats like SMART and TREC
• Parsing and tokenizing text
• Maintaining corpus statistics (term frequecies), creating inverted indexes
• Creating weighted document vectors (TF--IDF) based on corpus statistics

Most of the code is in Python, with some crucial functions implemented in C++.

Getting started

The atoms sub-package provides all functionality related to representing text documents as numbers. This is a good place to start using the Mekano package.

The ml sub-package provides access to classifiers and related utilities.

The dataset module provides a handy class for representing and working with datasets (collections of documents).

The io module contains several functions for reading common file formats and working with Python pickles.

The evaluator module contains evaluation tools.

The textual module contains functions for parsing and tokenizing text.

The indri module provides a simple interface to the Indri binaries.