anki-rwkv · factory overview · model architectures · R01 campaign · 8k/2k run · data quality

Factory overview: what runs, what is planned, what is done

The whole anki-rwkv work in one place. The page updates itself from the job queue and the scoring files; takeaways and the backlog are edited by hand. Today it holds one track, the small model: a 331,511-parameter student (64 channels, distilled from the big model, the fork’s 2,762,884-parameter RWKV) trained on 5,000 dataset users and scored on users it never saw, full history, both models with int8 weights and int8 state. LogLoss is lower-is-better; the % is the gap to the big model on the same reviews (positive = student worse). Target: ≤ +1.0% on both heads over the 464 held-out users.

1. Running now

2. Planned

Times marked ~ are estimates, laid out in your browser from the running job and each job’s expected minutes. A GPU job starts only if it also ends before the break starts; otherwise it waits until the break ends.

3. Backlog

Ideas and follow-ups with the reason they are there. Pointers from outside research are marked; we keep our own training setup.

4. Results

Newest first, GPU and CPU work together. Each row gives the absolute LogLoss (curve / instant), the gap to the big model on the same reviews, and the evaluation size. “Decay” = the final pass with the learning rate brought to zero; “window” = consecutive reviews of one user per training sample.