I built Contextrot to ask a narrow question: what happens to verifiable facts when a model repeatedly summarizes its own previous summary?
Design decisions
The harness starts with fifty invented facts in five categories. A deterministic Python grader distinguishes facts that are kept, lost, or corrupted after each cycle. Invented facts reduce the chance that a model can reconstruct missing details from general knowledge. Every summary is committed, so analysis can be rerun without a model server.
The grader needed debugging too
Inspecting the raw text exposed false losses caused by Unicode hyphens and Markdown emphasis. Normalizing punctuation and markup corrected the measurements. A seed self-check now refuses to start unless the grader finds all fifty original facts.
The repository includes transcripts, grading code, unit tests, and a verification script. The published curves show the recorded experiment rather than a live benchmark.
Limits
This is one seed document, one summarization prompt, and three local models served by Ollama. It does not establish universal model behavior or measure whether a model can act on surviving facts. The lenient rewording list was created after inspecting transcripts and only covers negations; it cannot support a fair comparison across categories. The useful lesson is to inspect and test the measurement itself.