Storage operations

NYXDB writes accepted changes into a committed tail, flushes durable table state into immutable NYXP parts, and rewrites parts through background compaction. Queries overlay the committed tail on eligible parts so recently committed rows remain visible before the next flush.

Select a table policy

Every CREATE TABLE names a policy:

CREATE TABLE trades (
  id UInt64 NOT NULL,
  market String NOT NULL,
  amount UInt64,
  PRIMARY KEY (id)
) SETTINGS
  mode = 'append',
  storage_policy = 'disk_data';

Use ephemeral_memory only when loss on restart is intentional. memory_data serves from memory but is durable when the process has a data directory. disk_data and sync_disk use durable local-disk parts; sync_disk also requests synchronous policy WAL.

Observe parts and placement

SELECT part_id, relation, shard_id, rows, size_bytes,
       min_key, max_key, disk, created_at_ms
FROM system.parts
WHERE relation = 'trades'
ORDER BY created_at_ms DESC;

SELECT name, pool, path, role, used_bytes, max_bytes, parts,
       node_id, type
FROM system.disks;

SELECT name, serve_pool, durable, delta, wal, compaction, ttl,
       tables_using
FROM system.storage_policies;

Alert on capacity, missing/unavailable resources, sustained part growth, degraded policy health, and a committed tail that does not drain.

Flush controls

The executable exposes relation-level row/byte threshold overrides, an age trigger, and a flush polling cadence:

• --flush-rows
• --flush-bytes
• --flush-after-ms
• --flush-poll-ms

Use policy defaults until production-like tests show a reason to override. Smaller thresholds reduce tail size and recovery work but create more parts and write amplification. Larger thresholds reduce flush frequency but consume more memory and increase tail overlay/recovery work.

Committed-tail row/byte caps and the delay ramp bound what happens when ingest outpaces flush. See Memory governor.

Compaction

Compaction merges immutable parts, applies retention/TTL behavior, and performs eligible policy movement. Principal controls include:

• --compact-poll-ms, with 0 disabling compaction;
• --compaction-jobs;
• --compaction-max-read-mbps and --compaction-max-write-mbps; and
• --compaction-io plus the direct-I/O fallback/strict policy.

Disabling or under-provisioning compaction can cause part count, read amplification, and disk usage to grow. Over-provisioning it can steal I/O from foreground reads and WAL/flush work. Establish baselines for both foreground latency and part convergence.

Experimental local topology

Resources, pools, and policies can be declared through storage topology DDL. The community runtime accepts local memory and local-disk resources with node_id = 'local'. jbod and mirror describe local placement; they are not cross-node HA.

For a resource change:

1. confirm the new mount exists and has correct ownership/capacity;
2. add the resource;
3. observe placement and policy health;
4. drain the old resource;
5. verify no required part remains; and
6. remove it.

Never remove a mounted data root underneath the running process.

Cache and I/O posture

--storage-cache-profile provides query, balanced, and ingest starting points. Explicit metadata, decoded-chunk, bitmap-row, materialized chunk, and retained-file limits override the profile when they appear later on the command line.

--storage-io=auto selects the platform path; Linux may use io_uring. Treat an opt-in backend or mmap mode as a release/workload change and verify it on the exact kernel, filesystem, mount options, and storage class.

Foreground read cancellation

One-shot cancellation and peer-disconnect checks propagate through the complete foreground read path, not only the top-level SQL operator. The current release checks cooperative stop state while reading:

• immutable persisted parts and flushed row-store data;
• committed in-memory tails;
• prepared/index reads;
• attribute reconstruction and revision overlays; and
• parallel shard selectors.

Parallel readers share a sticky, thread-safe peer state. One worker probes the connection without blocking its peers; contending workers keep progressing to their next checkpoint, and the owner rechecks after every worker joins so the stable cancellation reason wins over a generic selector error.

This is bounded cooperative cancellation, not asynchronous interruption of an arbitrary blocked filesystem or device syscall. --query-checkpoint-interval defines the maximum work units between engine checks. Raw TCP FIN/reset and WebSocket Close produce NYXDB_EXEC_CANCELLED: client disconnected; explicit wire/SQL cancellation produces the query-cancelled reason. Both are retained in system.query_log with completed scan counters.

During rollout, test cancellation against the actual storage policies and shard count used in production. A memory-only smoke test does not prove that a flushed multi-shard scan releases readers, admission, and query memory correctly.

Capacity model

Reserve space for:

• live immutable parts;
• WAL and checkpoints;
• committed-tail spill/sealed artifacts when enabled;
• one or more concurrent compaction outputs;
• topology movement; and
• restore targets and operational diagnostics.

The backup repository must not share the live data directory. Current native backup v1 covers the synthesized default disk only; customized multi-root or mirror placement requires a storage-level procedure covering every configured root until a matching backup backend is available.