Skip to content

Ingestion, coverage and the retained-ingestion manifest

VerifiedOwner Backend / platformLast verified backend@efa38e04e2db37a444e9f17100323c2ad4b83647 nautilus@2.0.0rc3

Ingestion in ONIX means: for a given venue integration, which instruments are subscribed on which channels, whether they are actually delivering data, and — separately — which instruments are allowed to accumulate durable demand at all. These are tracked by two different mechanisms: a per-shard coverage ledger, and an operator-controlled retained-ingestion manifest.

Coverage accounting

The completion contract for a full-universe integration is treated as an accounting identity, not a vibe: every instrument the venue discovers must end up classified, and the totals must reconcile with discovery. CoverageLedger is that ledger, one instance per spine process, written by the Nautilus-thread actor and read by gateway threads through GET /v2/coverage.

Vocabulary, verified in source:

  • status (per instrument) — ACTIVE, or INACTIVE:<venue-reported reason> at discovery time. Absent classification hook = every discovered instrument counts ACTIVE.
  • eligible (per instrument+channel) — whether the instrument's product family claims the channel at all. An ineligible channel (e.g. external bars on an options product a venue doesn't serve them for) is reported honestly as upstream_unsupported, never silently skipped.
  • planned / subscribed / failed — the subscribe call's own lifecycle on this shard.
  • with_flow / quiet — a subscribed channel that has, or has not, delivered at least one event. Quiet is not failure. A thin instrument may legitimately not trade for hours; it is reported quiet so nothing hides in an unreported tail.

One channel — internal_bars, the INTERNAL-aggregated tier described below — is marked sparse_ok: true in the coverage snapshot: a minute with no trades legitimately produces no bar, so quiet flow on that channel specifically is expected sparsity, not a missing subscription.

GET /v2/coverage/failures and GET /v2/coverage/quiet expose the failure and quiet sets directly, and GET /v2/coverage/{instrument_id} gives one instrument's full detail — all 404 before the actor has registered discovery, never a fabricated empty snapshot.

Gaps, duplicates and out-of-order data

Regressions in trade sequencing are counted explicitly (an out-of-order counter increments on the actor's own state whenever a later-arriving trade's timestamp precedes what was already seen). For derived bars specifically, an unresolved gap in the durable 1-minute series is tracked as a first-class interval; the derivation layer consults that gap ledger before building any derived bucket and withholds a bucket whose constituent window intersects an unresolved gap, rather than serving a value computed from an incomplete window (see aggregation.md). Nautilus itself performs no bar-level deduplication — two bars with identical timestamps are both delivered, in arrival order — so ONIX's own catalog write path (sorted, disjoint-range writes through the official catalog API) and this gap/duplicate accounting are what keep a re-delivered or out-of-order bar from silently corrupting a series.

Live 1-minute closes: two possible sources

A live 1-minute close can come from one of two places, and which one is active is a runtime configuration choice, not a fixed property of the codebase:

  • EXTERNAL — the venue's own candle, subscribed directly. Timestamp convention varies by venue (some venues stamp at interval start, some at close; ONIX profiles declare which, and start-stamped candles are normalized to close on ingestion).
  • INTERNAL — an ONIX-run collector subscribes to exactly one INTERNAL 1-minute BarType per instrument, which makes Nautilus's own DataEngine install a TimeBarAggregator over the trade stream the node already receives. This module performs no aggregation itself, constructs no Bar, and re-stamps nothing — it only persists what Nautilus's aggregator already produced. A minute with no trades produces no bar, live and persisted alike (no separate persistence-side filter), which is exactly the sparse_ok case in coverage accounting above.

Beyond bars: the generic custom-data ingress, kind by kind

Bars and trades are not the only data spine.actor.MarketSpineActor ingests. The same actor subscribes and projects six further Nautilus custom-data kinds through paired on_<kind> callbacks that each follow the identical shape — read the official Nautilus object, convert with a dedicated spine.wire.<kind>_to_wire function (never computed, always a straight field read), record it in coverage, then fan it out through SpineState:

Kind Actor callback Served via
Quote / trade ticks live-only subscription in on_start live stream only, never durable (see index.md)
Bars on_bar GET /v2/bars, /v2/stream/bars
Mark price on_mark_price GET /v2/facts?kind=mark_price, GET /v2/data/mark_price
Index price on_index_price GET /v2/facts?kind=index_price, GET /v2/data/index_price
Funding rate on_funding_rate GET /v2/facts?kind=funding_rate, GET /v2/data/funding_rate
Option greeks on_option_greeks (subscribe/unsubscribe gated on kind == "option_greeks") GET /v2/data/option_greeks
Instrument status on_instrument_status (subscribe/unsubscribe gated on kind == "instrument_status") GET /v2/data/instrument_status
Liquidation venue-specific (Binance: binance.wire.liquidation_to_wire, over the venue's own all-market stream) GET /v2/facts?kind=liquidation

GET /v2/facts (four kinds: mark_price, index_price, funding_rate, liquidation) and GET /v2/data/<kind> (funding_rate, index_price, instrument_status, mark_price, option_greeks) are two different read shapes over the same underlying latest-value slots — facts are live streams, never catalog rows, so a service without a given fact answers 404, never a fabricated empty page (spine.gateway_v2's own comment: "a service that has no facts... simply lacks the facts_latest method"). Both are additive: a node that never wired a kind up answers honestly that it does not serve it, rather than a silent empty success. Coverage counts an instrument_status observation the same way it counts a bar close — through the shared CoverageLedger described above — so a status/greeks/facts subscription that never delivers is visible as quiet, not invisible.

python/tests/test_gateway_v2_facts.py exercises the /v2/facts read surface directly against a constructed service; this documentation did not additionally confirm each on_<kind> actor callback against a live venue connection (see the observed-SSE caveat in ../runtime/index.md) — the ingress/projection code path itself is verified from source and the cited test; a live, end-to-end delivery of every kind from a real venue is unverified by this documentation pass.

The retained-ingestion manifest: production record versus development intent

Which instruments a spine is even allowed to accumulate durable demand for is governed by an operator-maintained retained-ingestion manifest, versioned and epoch-tagged. This is where the distinction between what is deployed and what is merely declared in the development tree matters most, and it is worth being precise:

  • The tracked production record (feature-state/rc3-data-plane-frozen-current-production.yaml, state CURRENT_PRODUCTION) states that the deployed manifest is mv5-epoch1, and names it — not the development roster below — as "the production instrument-demand authority." (production-record only; the live host was never independently observed by this documentation effort.)
  • The development-pin manifest committed in this repository (deploy/retained-ingestion.json) is a different, later version: manifest_version: 8, epoch: 1, covering Bybit (two active instruments, three historically retired at earlier manifest versions) and Hyperliquid (three active instruments). This file is what exists in the source tree at the pin — it is development intent, and nothing in the pinned repository proves it has been deployed.

These two documents are not the same manifest, and treating the development-pin file as "what production currently retains" would be a factual error. A concrete instance: the production record's own notes name CUDISUSDT-SPOT.BYBIT as part of the mv5-epoch1 production roster, while the development-pin's mv8 manifest already records that same instrument as retired (at manifest version 6, epoch 1). Both facts are true simultaneously — they describe the same instrument at two different manifest versions — and it is exactly the kind of drift a reader should expect between a frozen production pin and a development tree that has since moved on. See venues/instrument-identity.md for more on this identity-in-transition caveat.

What retention_state means on the wire

Every bar page for the durable 1-minute series carries a series_coverage envelope whose retention_state is authoritative from the manifest actually configured on that node:

retention_state Meaning
retained manifest active, instrument is in the resolved retained set
unretained manifest active, instrument is not in the set — zero ingestion demand; nothing will ever be written for it
unmanaged no retained-ingestion manifest configured at all (legacy namespaces)

history_terminal = exhausted OR retention_state == "unretained" is the one bit a chart client should terminate history paging on; a retained series with incomplete backfill reports history_terminal: false even if the current page is empty.

Evidence and source pins for this page

Verified. Current behaviour, confirmed in source at the pinned commit.

Verified on against the following immutable sources:

  • backend@efa38e04:python/src/onix_engine/spine/coverage.py
  • backend@efa38e04:deploy/retained-ingestion.json
  • backend@efa38e04:feature-state/rc3-data-plane-frozen-current-production.yaml
  • backend@efa38e04:python/src/onix_engine/spine/internal_1m_collector.py
  • backend@efa38e04:python/src/onix_engine/spine/actor.py
  • backend@efa38e04:python/tests/test_gateway_v2_facts.py

Status tokens are defined on the documentation and status model page. Every pin on this site is listed under versions and source pins.