Skip to content

Evidence and testing

VerifiedOwner Documentation integratorLast verified onix-docs@a2-publication

Documentation is tested here in the same sense code is: a claim is a hypothesis, and the review question is not "does this read well?" but "what would prove this wrong, and did anyone try?"

The evidence hierarchy

Sources are ranked. When two disagree, the higher one wins.

Rank Source Why it ranks here
1 Executable locks, pins, content hashes and schema manifests They are enforced by machinery, so they cannot silently lapse.
2 Source code at an immutable commit It is what actually runs, and it is checkable by anyone with access.
3 Tests at an immutable commit They demonstrate intent and pin behaviour, but only for what they cover.
4 Tracked records and manifests (for example a production record) They state intent authoritatively but are not observations.
5 Direct observation of a running system Strong for the moment observed, but perishable and rarely reproducible.
6 Repository prose — READMEs, design notes, handoff documents Frequently stale. Useful for intent, never authoritative for behaviour.

Rank 6 is the one that causes trouble. A confident README sentence describing behaviour that changed two releases ago is the single most common source of false documentation. Prose is a lead to follow, not a citation.

Note the deliberate ordering of 4 and 5: a tracked record is more durable evidence than an unrecorded observation, but an observation is stronger evidence of current state. This is why a page may carry Production record even though someone once saw the system working — a memory is not a receipt.

Evidence packets

Each documentation lane produces a machine-readable packet alongside its pages. The packet, not the prose, is what an independent reviewer works from.

{
  "agent": "backend-correctness-fixer",
  "pins": { "handoff": "efa38e04…", "functional": "f80bc316…" },
  "claims": [
    {
      "id": "be.data.catalog.durable",
      "page": "docs/backend/data/catalog.md",
      "claim": "One precise, falsifiable sentence.",

      "status": "verified",

      "availability": "implemented",
      "availability_note": "One sentence scoping where this is usable.",
      "availability_scope": ["core-spine"],

      "disclosure": "public",
      "disclosure_reviewed_by": "backend-correctness-fixer",

      "evidence": [
        { "path": "src/…", "blob": "40-hex Git blob id", "note": "what in the file supports it" }
      ],
      "confidence": "high",
      "falsifiable_check": "git -C <repo> show <pin>:src/… | grep -n '…'"
    }
  ]
}

Every field above is required. The loader is fail-closed: a claim missing any of them aborts the generator and therefore the build. Nothing is defaulted, and nothing is dropped quietly.

Three independent dimensions

status is provenance — how well the claim is known. availability is lifecycle — whether anyone can use the thing, and where. disclosure is publication — whether this exact sentence may appear on the public site.

They are deliberately never collapsed. verified + unavailable is a normal, important combination: the code was read at the pin, and the code refuses. A single badge answering all three questions is how a deferred capability ends up documented as a working one.

A small number of pairings are contradictory and are rejected mechanically: retired must be unavailable; target-contract must be planned or preview; a quarantined claim can never be public.

Blob ids are mandatory

Every evidence anchor carries a 40-character Git blob id, not just a path. A path names where something was; a blob names exactly what was read. Without the blob, a reviewer re-running a check a month later is reading a different file and does not know it.

The publication decision

An evidence packet is private working material. It legitimately holds guard names, authentication mechanics, storage locations and scope semantics, because its purpose is to let a reviewer with source access re-run a check. None of that is publishable copy just because the page it supports was not on a withheld list.

So each claim declares one:

  • disclosure: "public" — the sentence renders verbatim in the capability register;
  • disclosure: "internal" plus a public_claim — the separately reviewed, bounded rewording renders instead;
  • disclosure: "internal" alone — nothing renders, and the claim is counted as withheld so the register cannot pretend to be complete.

When in doubt, mark it internal. The register being smaller is a cost; the register leaking a withheld subject is a failure.

Four fields carry the weight

claim must be one sentence that could be wrong. "The catalog is robust" is not a claim; "the durable catalog stores one-minute bars and every coarser resolution is derived" is.

falsifiable_check must be a command, not a suggestion. If a reviewer cannot copy it and run it, the claim is not reviewable. A check that merely locates a sentence in a docstring is not a falsification check: prose is rank 6 in the hierarchy above, so a claim whose only check greps prose must be graded down rather than marked verified.

availability_note must scope the claim. "Available" with no scope is close to meaningless in a system where availability is genuinely per node.

confidence is the author's own honest grading, and low is an acceptable answer. A low-confidence claim that survives review is worth more than a high-confidence claim nobody tested.

Adversarial review

Review is conducted as an attempt to falsify, not as a read-through. An independent reviewer — in the current process, a separate agent with no write access to any documentation page — takes representative claims across every lane and runs their checks against the pinned sources.

The reviewer's job is to produce one of three outcomes per claim:

  • Refuted — the check contradicts the claim. The page must change before release.
  • Confirmed — the check supports the claim at the stated pin.
  • Not checkable as written — the check does not actually test the claim. This is a defect in the claim, not in the review, and the claim is downgraded to Unverified until it is rewritten.

That third outcome matters. A claim with a check that cannot fail is indistinguishable from an unsupported claim.

What is checked mechanically

The build performs these checks on every release, independent of human review:

  • every page has complete, valid provenance metadata, and exactly one <h1>;
  • every claim carries a status, an availability with scope, a disclosure decision and a Git blob id on every evidence anchor — a claim missing any of these fails the build rather than being defaulted;
  • every internal link and anchor resolves under strict mode;
  • every page declares the canonical URL the route contract assigns it;
  • every one of the 61 legacy pages resolves to a published route, a redirect, a merge destination, both halves of a split, or a recorded quarantine entry, with no silent drops — and rows that are published but still carry an open blocker are recorded as such rather than counted as closed;
  • generated pages still match the inputs they derive from, checked before anything is regenerated and into a temporary tree, so the check can neither miss drift nor erase it;
  • the drift gate is itself tested: a temporary hand edit must make it fail, and must survive the failing run;
  • the published page set, the search index and the sitemap match the route contract exactly — an extra entry fails as loudly as a missing one;
  • no forbidden brand term, credential-shaped string or private-topology path appears in published output;
  • no withheld subject is disclosed under a different route, checked semantically rather than by URL;
  • every tracked file in the repository matches an explicit allowlist, and no build output, virtual environment or cache is tracked;
  • two consecutive production builds produce identical bytes, with the raw and normalized comparisons both recorded.

Testing the documentation is not testing the system

A final caution. Everything above establishes that this site accurately describes source at a pinned commit. It does not establish that the system is correct, that it is deployed, or that it behaves as its source suggests under load.

Those are different questions with different evidence, and where this site cannot answer them it says so rather than borrowing confidence from the fact that its own process is rigorous.

Evidence and source pins for this page

Verified. Current behaviour, confirmed in source at the pinned commit.

Verified on against the following immutable sources:

  • onix-docs:tools/gen_reference.py
  • onix-docs:tools/evidence_model.py
  • onix-docs:tools/check_drift.py

Status tokens are defined on the documentation and status model page. Every pin on this site is listed under versions and source pins.