# Recall matrix

**Recall matrix** introduced in Algorithm SM-17 collects recall for all difficulty, stability and retrievability quantiles. This is an equivalent of 8000 data bins for plotting independent forgetting curves for all repetition populations. In the ideal case, recall registered in the recall matrix should correspond 1-to-1 with retrievability dimension. In practice, this is always hard to achieve even in quantiles with large numbers of repetition cases. Comparison of retrievability and recall values in the **recall matrix** can be used in algorithmic assessment with the help of the universal metric.

This glossary entry is used to explain SuperMemo, a pioneer of spaced repetition software since 1987

Figure:Recall matrixin SuperMemo is a 3-dimensional matrix with recall registered for all repetition populations separated by difficulty, stability, and retrievability. The matrix is used to predict retrievability at review and to assess the algorithmic performance. In the picture, repetition record for difficulty quantile 0.35 is presented. Horizontal axes represent stability and retrievability. The vertical axis shows recall in repetitions. 11,943 repetition cases are presented. Quantiles with less than 10 repetitions have been rejected as outliers

Universal Bet-Win metric can be used to assess the algorithmic performance in accurately predicting recall:

Figure:A universal Bet-Win metric in SuperMemo shows the areas of stability and difficulty where predictions might be less accurate. The universal metric is a square root of the mean square of the B-W metrics for individual retrievability quantiles. Red color indicates poor estimates for small populations. White and bluish colors indicate accurate predictions. Columns group items by difficulty, while rows correspond with stability categories (expressed in days). Averages on the right and at the bottom are not arithmetic correspondents of their rows and columns due to weighing up by repetition counts. The ultimate metric for the entire data set can be found at the bottom-right corner, and shows that there is still room for improvement by parameter optimization. The ultimate goal is to fine-tune the algorithm to get to the universal B-W metric approaching zero for large datasets.Important!The actual metric in SuperMemo will be better than the one shown in the bottom-right corner. This is due to the fact that the presented metrics have been derived from the Recall[] matrix that is based on the theoretical retrievability (theory vs. data), while the ultimate universal metric in SuperMemo will use best weighted retrievability prediction that includes the Recall[] data (data-based prediction vs. grades)