We outline below the toolkit's functionality by following through a concrete example applying the toolkit in a Matlab environment. The example below demonstrates the usability of the toolkit in the different stages of predicting and evaluating collaborative filtering.

The first step is to read the data set and store it in memory. The toolkit may be used to parse EachMovie data and read it to memory by issuing the routine eachMovieReader:


The number 300 represents the number of users that are read from the dataset. The dataset contains data gathered from 80000 users - to read all of them replace 300 by 80000. This the entire dataset should take several minutes and exhaust a large part of the physical memory. Reading a subset of 300 users should a few seconds.

The output userVoteMat is a sparse matrix in which userVoteMat(i,j) is the recorded preference of the user i for movie j. If no preference with respect to that item was reported by the user the entry will be 0. To view the preference of user i type userVoteMat(i,:) in the matlab prompt. To view the sparsity pattern of the matrix type spy(userVoteMat).

The next step is to split the data set into a set of active users and a set of non-active users. The latter set will be used to provide predictions for the active users preference. This division is analogous to a training set/testing set division in classification scenario. The division basically chooses some of the rows randomly and form a new matrix activeMat with the remaining rows forming the new matrix otherMat. This may be done using the routine splitUsers below


The second and third arguments in the function call are the numbers of active and non-active users respectively. Other numbers may be selected of course, but the sum of both numbers should equal the number of users. Note that userVoteMat may contain less users than 300. This is because the routine eachMovieReader post-process the users and removes users with extremely low number of preference. For example, users with a vote of only one item are completely unusable in a collaborative filtering framework.

The next step is to partition the set of items to observed and predicted items. The first set is the set of items that the active user reports his preference on to the system. The system is then trained based on that information. The second set of items that was held out for the training phase is used for evaluating the system's responses. The system recommendations for the second set of items for the active user is compared to the active user preference and using one of the evaluation measures, a numeric score is calculated. Partitioning the items is done inside the routine EachMovieComparison. To perform it manually outside the routing, simply cut and paste the code to the Matlab environment.
To obtain a prediction of a user's preference use the routines predictPreferencePD for personality diagnosis prediction and predictPreferenceMemBased for memory based prediction. For example,

>>[pred,meanPred]=predictPreferencePD(activeUser, otherUserMat,numValues,sigma)

returns the probability of different preference for different items (a stochastic matrix) in pred and the mean predicted preference (row vector) for each item in meanPred. activeUser is a row vector denoting the preference of the active user on the reported items and otherUserMat denotes the preference of the other non-active users. numValues is the number of different preference values (6 in EachMovie dataset) and sigma is the standard deviation of the Gaussian distribution in the PD model. This is a free parameter and requires some tuning for optimal performance.

The prediction output from predictPreferncePD and predictPreferneceMemBased may be evaluated using one of the evaluation metrics as described in the toolkit description. For example, the routine rankedEvalCF evaluates the given recommendations by ranking the items according to the output mean predicted values and then computing the expected utility of a user browsing the list, with the probability of the user selecting an item on the list decays exponentially with the rank of the item in the list.
It is also possible to evaluate the methods directly, without calling predictPreferencePD or predcitPreferenceMemBased by using the supplied routines evalPDEachMovie,  evalMemBasedEachMovie, evalConstEachMovie, evalAvgEachMovie with each routine returning the evaluation L1, L2 and ranked evaluation measures for different prediction strategies.

For example,


returns the L1 error, the L2 error and the ranked evaluation measure for the memory based method described by simMethod.