Storage model

How committed tails, local WAL, immutable NYXP parts, storage policies, and compaction provide single-node persistence.

NYXDB storage is local and shard-owned. A query reads a coherent union of the committed tail and immutable NYXP parts; flush and compaction change the physical shape without changing the committed result.

Commit and publication

For durable DML, the host:

validates the statement and takes the target relation's publication fence;
captures a rollback preimage and stages the mutation;
appends data records and a trailing commit manifest to the local journal;
on success, publishes the store heads, emits PSI events, and acknowledges;
on failure, restores the store and physical journal tail and returns an error.

The exact fsync boundary depends on the storage policy and --wal-sync. An asynchronous journal append is not equivalent to power-loss durability. See Durability.

Committed tail

Newly committed rows are immediately queryable from the in-memory committed tail. Disk-backed tables maintain pruning metadata for that tail so a query does not have to materialize every unflushed row.

Capacity, delay, sealing, and optional spill behavior are bounded by server configuration. Under the default mixed-ingest profile, a full tail waits for capacity and eventually rejects the insert as retryable. For eligible disk-resident, non-keyed append tables, the bulk profile uses non-blocking admission with sealing and spill to keep the writer moving. Keyed and memory-resident tables retain enforced-cap backpressure because they have no equivalent drain path.

Immutable NYXP parts

Background flush turns sealed data into immutable NYXP parts:

columns are stored in typed, compressed chunks;
parts are sorted by the table's order key;
metadata and index regions are covered by checksums;
publication uses a temporary file, fsync, and atomic rename;
readers use immutable snapshots while new parts publish.

Part files are local. Object-store parts and archival are Unsupported in the community runtime.

Compaction and retention

Compaction merges parts, applies keyed latest-wins collapse and tombstones, rebuilds index regions, evaluates TTL, and performs configured pool movement. It runs on a background cadence and can be I/O-rate limited.

TTL is a predicate string evaluated lazily during compaction:

CREATE TABLE events (
  id UInt64,
  ts DateTime64(3),
  payload String
) SETTINGS
  storage_policy = 'default',
  ttl = 'ts < now() - INTERVAL 30 DAY';

TTL is not an immediate row timer. Data remains visible until an eligible compaction applies the predicate.

Built-in policies

The engine synthesizes these policies when the topology does not override them:

Policy	Serving and durability behavior
`ephemeral_memory`	Memory-serving, non-durable rows; DDL can persist with a data directory, but rows reset on restart and never flush.
`memory_data`	Memory-serving and durable; WAL and immutable artifacts use the local default disk.
`disk_data`	Disk-serving durable parts with memory-resident index policy.
`sync_disk`	Disk-serving and fsync-before-ack for DML.
`default`	Durable disk-serving policy over the synthesized local default disk.

memory_data is not ephemeral. Use ephemeral_memory only when losing all table rows on restart is acceptable.

Storage topology

Local resources are grouped into homogeneous jbod or mirror pools, then referenced by policies. Mirror placement is local write-all redundancy; it is not cross-node replication or high availability.

Topology DDL and multi-pool movement are Experimental. Only node_id='local' and local disk/memory resources are accepted. See Storage operations for YAML import, live DDL, drain, and lifecycle guards.

Inspecting physical state

SELECT relation, shard_id, rows, size_bytes, disk
FROM system.parts
ORDER BY relation, shard_id;

SHOW STORAGE RESOURCES;
SHOW STORAGE POOLS;
SHOW STORAGE POLICIES;