concept co - occurrence in category machine learning

This is an excerpt from Manning's book Mahout in Action. Login to get full access to this book.

Something like this is needed for the algorithm: a co-occurrence matrix. Instead of computing the similarity between every pair of items, it’ll compute the number of times each pair of items occurs together in some user’s list of preferences, in order to fill out the matrix. For instance, if there are 9 users who express some preference for both items X and Y, then X and Y co-occur 9 times. Two items that never appear together in any user’s preferences have a co-occurrence of 0. And, conceptually, each item co-occurs with itself every time any user expresses a preference for it, though this count won’t be useful.

Co-occurrence is like similarity; the more two items turn up together, the more related or similar they probably are. The co-occurrence matrix plays a role like that of ItemSimilarity in the nondistributed item-based algorithm.

6.3.3. Translating to MapReduce: calculating co-occurrence

The next phase of the computation is another MapReduce that uses the output of the first MapReduce to compute co-occurrences.