Durable catalog¶
ONIX's durable market-data store is a thin, typed façade — CatalogStore — over Nautilus's own
nautilus_trader.persistence.ParquetDataCatalog. Per the backend's own governing rule ("use
Nautilus v2 off the shelf"), this layer adds no persistence logic of its own: no custom Parquet
layout, no compaction, no sharding, no archival, no object store. Sorting before write, disjoint
range enforcement, on-disk layout, and round-trip field fidelity are exactly what
ParquetDataCatalog already provides.
One catalog authority, two physically separate roots¶
ParquetDataCatalog is the sole persistence mechanism — but this backend runs it against two
physically separate roots on disk, never relabelled and never cross-filled, per the repository's
own canonical "market-data persistence and inode lifecycle" statement: an EXTERNAL root holding
venue klines (arrive only via the official adapter's REST request_bars, stamped
…-1-MINUTE-LAST-EXTERNAL, history/backfill/repair only) and a separate INTERNAL root holding
the live-derived series (…-1-MINUTE-LAST-INTERNAL, from the official DataEngine
TimeBarAggregator over TradeTicks — see ingestion.md). One CatalogStore
API serves both; which root a given write or read touches is a matter of which bar-type suffix and
namespace it names, never an implicit default.
Both roots are consolidated through the same pinned, public Nautilus API
(consolidate_catalog/consolidate_data/consolidate_catalog_by_period/
consolidate_data_by_period; the *_by_period fixed-window default is one day,
period_nanos=86_400_000_000_000 — a Nautilus-native roughly-monthly 30-day option also exists
and is explicitly not a calendar month or a Nautilus recommendation), but on different schedules:
the rolling EXTERNAL catalog is consolidated by the spine's own maintenance timer
(spine/catalog_maintenance_isolated.py); the INTERNAL sibling root has its own, separately
scheduled closed-period consolidation (spine/internal_lifecycle.py) that runs only over closed
periods (watermark plus a grace window), never touching the still-open period a live write is
landing in.
Why this became a fail-closed admission concern, not just a scheduling detail. A 2026-08-24
production outage exhausted 4,915,200 of 4,915,200 inodes on a 149 GiB volume that was still
roughly half empty by bytes — small-file amplification from an unconsolidated tree (the retired
Binance rolling tree alone held 2,443,223 files), not byte capacity, was the binding constraint.
The deployed release's response, ONIX_OPERATIONAL_CONFIGURATION layered beside Nautilus's own
mechanism (never a modification to it): writers to the INTERNAL root are admission-gated on
min_free_inodes/max_files floors, and every consolidation/hydration cycle records census,
watermark, and error telemetry so a starved filesystem is visible before it becomes an outage
rather than after. A later, source-committed revision of this same lifecycle additionally adds a
min_free_bytes floor and makes the file-count figure observational metadata only — an arbitrary
file-count ceiling is never itself what refuses a live write. Deployment as a whole is separately
required to fail closed: application code resolves catalog roots from configured paths
(ONIX_DATA_ROOT and per-integration namespaces) rather than hard-coding a device or mount point,
and a catalog-writing service must refuse to start if its expected volume is not mounted and
writable — a missing mount is a startup refusal, not a service that starts and silently writes
into whatever local path happens to be there.
The retained-ingestion manifest, not a canary roster, is the production instrument-demand
authority. deploy/retained-ingestion.json (see ingestion.md for its versioned
epoch structure) is stated in source as the authority for which instruments either catalog root may
accumulate durable demand for; an earlier 25-instrument canary roster is retained only as evidence
history, never re-read as a current authority.
What CatalogStore narrows¶
ParquetDataCatalog exposes roughly forty methods; CatalogStore narrows the surface each ONIX
caller actually needs: write_instrument, write_bars, write_trades, and their read_*
counterparts, plus a filesystem-only measure() used for status telemetry. Callers that need
anything else reach .catalog, the underlying official object, directly — nothing is hidden, the
narrowing exists only to keep call sites simple. Writes are pre-sorted ascending by ts_init
before every call, because the official writer requires that order and raises otherwise; this is
calling the official API correctly, not adding a second ordering rule.
Retention shape¶
Durable retention is instruments plus, at most, the latest 5,000 closed 1-minute bars per
instrument — a rolling window, not an unbounded archive. Quotes, trades, and per-second bars flow
live (streams, counters, watermarks) but are never durably stored. Every resolution other than the
durable 1-minute series is derived from that window on read (see
aggregation.md); a request for history outside the retained window returns an
honestly empty page rather than a fabricated one, and the wire-level series_coverage envelope
states explicitly whether a series' backfill is exhausted (complete) so a client knows when to
stop paging.
GET /v2/status's catalog block (root, files, bytes, measured_ts_ns) comes from a
low-frequency background filesystem measurement, not a per-request walk — measured_ts_ns: "0"
means "not yet measured," not "empty."
Hydration: bringing a series up to the rolling cap¶
Getting an instrument's rolling window populated is a scheduled backfill, not a synchronous call:
DataActor.request_bars(...) results always arrive later, through on_historical_bars, and
HydrationManager is the bounded, priority-ordered scheduler that drains that queue. It owns no
Nautilus objects — the one I/O it performs is persisting its own retry state to a single JSON
document, atomically, so classification and retry eligibility survive a process restart.
Status vocabulary¶
The scheduler's internal state machine has six states: EMPTY → HYDRATING → READY | NO_HISTORY |
RETRY_WAIT | FAILED, where RETRY_WAIT and NO_HISTORY re-enter HYDRATING once their next
eligible time passes (backoff for the former, a slow re-check cadence for the latter — a delisted
instrument stays empty forever, an illiquid one may trade tomorrow). The governed GET
/v2/hydration wire contract publishes a four-value status — EMPTY | HYDRATING | READY |
FAILED — per instrument, alongside count, oldest_ts_ns, newest_ts_ns, and last_error;
NO_HISTORY and RETRY_WAIT are internal scheduling detail, not part of the published wire
vocabulary. count, oldest_ts_ns and newest_ts_ns are read directly off the catalog, so
"the wire says READY" and "the catalog has rows" cannot disagree.
FAILED is terminal for the scheduler but not for the world: a gateway request naming the
instrument (promote()), or a live close durably observed for it (record_append()), both
requeue it immediately — the instrument evidently trades now.
Why "timeout" and "no history" are kept apart¶
An earlier deployed run left hundreds of instruments FAILED with a generic timeout message. The
underlying spine log showed the real split: a genuine HTTP-level failure aborts the request before
any response is sent (a true timeout, transient), while a successful request with zero candles
delivers an empty response the actor sees as on_historical_bars([]) — a stable "no history"
answer, not a failure. Because the pinned adapter's Python callback carries no request id back to
the caller, HydrationManager correlates an empty answer to its instrument by elimination
against the small set of requests genuinely in flight, using the catalog itself as ground truth
(rows on disk after the request means "gap was simply empty"; no rows means "confirmed no
history"). This is why the in-flight request count is deliberately kept small — a large in-flight
map makes an empty answer's owner ambiguous, and ambiguity was measured to cost real hydration time
by spending retry budget instruments never needed.
Never re-requesting a complete series¶
next_to_start() returns a start hint alongside the instrument id: a series that already has bars
requests only the gap above its own newest close, never the full window again. A process restart
therefore costs one small gap request per already-READY series, not a full rehydration of the
whole universe.
Related pages¶
aggregation.md— how every non-1-minute resolution is derived from this catalog.ingestion.md— subscription/coverage accounting and the retained-ingestion manifest that decides which instruments this catalog is even allowed to accumulate demand for.
Evidence and source pins for this page
Verified. Current behaviour, confirmed in source at the pinned commit.
Verified on against the following immutable sources:
backend@efa38e04:python/src/onix_engine/catalog/store.pybackend@efa38e04:python/src/onix_engine/spine/hydration.pybackend@efa38e04:README.mdbackend@efa38e04:docs/contracts/market-spine-v2.md
Status tokens are defined on the documentation and status model page. Every pin on this site is listed under versions and source pins.