1. One history, five timelines
A made-up user reviewing five cards over two weeks. Each square is one review, coloured by card, with the grade inside (A = Again, H = Hard, G = Good, E = Easy). Pick a stream to see which reviews it links together, then click a review.
2. The memory: a small matrix that learns while it reads
Inside each stream, every head keeps a state matrix S. Each review writes a value at a key; a later review reads the matrix with its own receptance vector and gets back what was stored near it. This toy uses 4×4 instead of 32×32 and four slightly overlapping keys, one per card, and reads each card back with its own key.
State S (rows: value dims, columns: key dims)
Reading each card’s key back
Bars: what S returns for that key. Black ticks: the last value written for it. Right: the error.
3. From memory to a forgetting curve
The output head does not predict a single number. It predicts a whole forgetting curve, built as a weighted mix of simpler curves. The models on this site build that mix in two different ways; switch between them. Hover the chart to read a probability.
0.9t / s with a fixed stability s; the model only chooses the weights. Drag the bars to set them (they are normalised to sum to 1), or try a preset. Mixing fast and slow curves gives a long tail that a single exponential cannot: a card can be partly forgotten within a day yet keep a stable core for months. The real heads use 128 (Alex) or 64 (the older compact-model iter36 config trained in this repository) basis curves from 0.1 s to about 100 years, and add a learned correction at the same number of time points. The compact model from outside research does not work this way: see the learned power-law curves.RWKV_GRU_HEAD=3, modelled on srs-benchmark’s GRU) use only 3 curves, but predict each one’s weight, stability S and decay d for every review: R(t) = Σ wi·(1 + t/Si)−di. That is a power-law shape, like FSRS’s forgetting curve, so a single curve already has a long tail. There is no correction head, so predicted recall can only fall over time. Dashed lines are the three curves; the solid line is their mix.Either way, a separate Instant head predicts the four grades for the review that is about to happen.
Putting it together
- Inputs. Each review becomes 92 numbers: intervals, the grade, the answer time, today’s workload, day cycles and random ID tags. See the architecture page.
- Streams. The encoded review passes through the card stream, then deck, note, preset and user streams. Each stream is a stack of RWKV-7 layers whose memory is shared only among reviews with the same ID.
- Two questions per review. A query copy with the grade hidden predicts the grade (Instant); its update to memory is discarded. The real review then writes to memory and predicts recall at the next review’s delay (Curve).
- Training. All of this runs in parallel on the GPU over thousands of users’ histories. The R01 campaign made that training several times faster without changing a single bit of the result.