Skip to content
Archive
No. 010AI SystemsProtocol

Qualifying a Model Council: Eight Falsifiable Laws

Why a board of advisors is usually one model in several costumes, and how to prove otherwise before it ships

Mixture-of-Agents councils are usually assembled as a taste decision, which produces rosters that fail without error: advisors cut off mid-sentence, one model advising itself, a weak synthesizer, models the API cannot call, and a bill nobody predicted. This paper presents a qualification protocol of eight falsifiable laws, each derived from a defect met in production, with measurements. The same question cost 1.18 dollars in an aged session and 0.0144 dollars in a fresh one, and a more densely encoded payload cost 2.7 to 4.4 times the tokens. A dated field-notes section records what the protocol caught, including defects that shipped past review.

Date
AUG 2026
Read
11 min
Depth
Depth: DeepDeep
Cluster
AI Systems
Department
Research Engineering
Provenance
Third Signal skill packs · public repository · v1.3.0
Authors
Third Signal Lab
6:59
AI-generated explainer, made from this paper with Gemini Notebook. The paper is the source of record.
About 11 min left

Thesis

A council nobody qualified is usually one model in several costumes.

Model selection for a council is treated as taste: pick the leaderboard leaders and ship. The result is a roster that is quietly broken in ways no test catches, and that returns a plausible answer at a higher cost with fewer real perspectives than the bill implies.

01 · Argument

Eight falsifiable laws, each from a real defect.

Catalog, reasoning-token, recency, fan-out, unit-price, sensitivity, context and compression. Each contradicts an intuition most teams hold, and three were caught only after the roster had already shipped and passed review.

Evidence

The same question cost 1.18 dollars in an aged session and 0.0144 in a fresh one.

Input to output tokens ran 130 to 1 in a 12-day session and 3.1 to 1 in a fresh one, because every advisor receives the whole conversation. The first estimate for a live turn was about 25 times too low.

Diagram · Role determines the metric

Advisors are ranked on domain axes and lineage diversity. The aggregator is ranked on instruction following and, if it drives tools, agentic skill. The same seats appear in agent swarms: workers, an adversarial verifier from a different lineage, and a synthesizer.

Signal 01Weight 5 of 5Proposed
“A council of four models from one family buys correlated errors at four times the price.”
Signal 02
“Output caps count reasoning tokens, so a reasoning model can spend its whole budget thinking and emit a fragment as advice.”

Tension

Static qualification is necessary and not sufficient.

The roster was qualified and validated, and still shipped with defects found only by running a real turn: a cap calibrated on a toy prompt, a cost estimate 25 times low, and a verifier that reported a false pass.

Implication

Qualification is adversarial or it is theatre.

An early confident conclusion, that a broker had no credentials, was checkable and wrong. Pushback corrected it and surfaced two existing skills that made half the original plan redundant.

Provenance

Derived from production qualification of a live operator-agent council. The field notes are dated 2026-08-05 and will age. The laws are written not to.