Skip to content
Archive
No. 009GovernanceBenchmark

GREATGAME: A Canon-Consistency Benchmark

Measuring whether a system can tell a contradiction from a supersession

GREATGAME is a benchmark specification for automated contradiction and supersession detection over the Doyle Sherlock Holmes corpus. Its central test is not finding contradictions but distinguishing an unintended contradiction from a deliberate, in-world supersession, because a system that flags both identically buries every ratified canon change in false alarms. It defines four instance classes, three separately scored tasks (extraction, detection, end to end), a headline Supersession Discrimination Rate, and an annotation protocol with two blind annotators. As of this draft, no instances have been annotated and no result exists.

Date
JUL 2026
Read
6 min
Depth
Depth: DeepDeep
Cluster
Governance
Department
Research Engineering
Provenance
Datum protocol repository · CC-BY-4.0 annotations · Draft 0.1
Authors
Third Signal Labs
About 6 min left

Thesis

The test is not finding contradictions. It is telling a contradiction from a supersession.

Any system that flags aggressively scores well at finding contradictions. A system that treats a deliberate canon change the same as an error buries every ratified change in false alarms, and gets uninstalled.

01 · Argument

Four instance classes keep the benchmark honest.

Conflict, supersession, apparent contradiction that resolves on inspection, and uncovered. Without the apparent class a benchmark rewards over-flagging, the exact behavior that makes continuity tools unpopular.

Evidence

Three tasks, scored separately: extraction, detection, and end to end.

Scoring only end to end hides which component is broken. The gap between detection on gold claims and the end-to-end score is the extraction risk, quantified. The headline metric is the Supersession Discrimination Rate.

Diagram · Corpus and instance schema

The full Doyle canon, four novels and 56 stories, public domain, pinned and checksummed. Each instance cites character spans and records whether the narrator believed the assertion, which is what lets a system reason about supersession.

Signal 01Weight 4 of 5Proposed
“An overall recall figure is a number. Recall by hazard is a finding.”
Signal 02Weight 3 of 5Proposed
“Name the benchmark's limitations before a reviewer does.”

Tension

The benchmark exists before any result does.

The datasheet states that no instances are annotated yet and ships empty on purpose. The release checklist is unchecked and no Supersession Discrimination Rate has been measured. The honest reading is a measurement plan with its dataset still to be built.

Implication

A canon-governance benchmark needs clean provenance on its own sources.

Knox, Sayers, Baring-Gould and Klinger are under copyright, so every treatment is paraphrased and attributed, and any verbatim text counts as a defect in the dataset.

Scholarly sources

Knox 1911, Sayers 1946, Baring-Gould 1967, Klinger 2004 to 2005, and the Baker Street Journal.

  1. 01Studies in the Literature of Sherlock Holmes (1911)
  2. 02Unpopular Opinions (1946)
  3. 03The Annotated Sherlock Holmes (1967)
  4. 04The New Annotated Sherlock Holmes (2004)