Production checklist

This checklist is intentionally concrete. Record an owner, evidence link, and review date for every item.

Day zero: establish the boundary

• The workload fits one NYXDB process and does not require replication, automatic failover, rolling upgrades, or multi-statement transactions.
• Every Experimental dependency has an explicit acceptance decision and fallback.
• Direct application endpoints, auth providers, query-driven limits, and operator access governance each have a separate Experimental acceptance decision when used.
• The exact image digest or binary Git revision is pinned; mutable tags are not the deployment identity.
• Server and client revisions have passed a protocol smoke test together.
• Every operator client used with --require-auth actually sends operator OP_AUTH on each socket. The current web console does; the current nyxsql and JDBC revisions do not expose operator credentials.
• The listener is reachable only on the intended closed network. Any browser or otherwise untrusted path uses a reviewed TLS-terminating edge with explicit Origin and network policy; native TLS is not assumed.
• The data volume, backup repository, and restore target are independent failure domains where the available infrastructure permits it.
• Capacity estimates include committed tail, parts, WAL, compaction headroom, query memory, and backup retention.

Before first start

• --data-dir points at a persistent, writable, dedicated path. Starting without it is intentionally in-memory.
• The container/process memory limit and --memory-limit-bytes agree.
• The memory governor remains enabled and its ramp/block behavior has been load-tested.
• WAL sync mode and every table's storage policy meet the written acknowledgement/RPO contract.
• Connection, query admission, queue, timeout, output, operator-memory, and cardinality limits are recorded.
• --query-global-exact-memory-bytes and its effective startup value are recorded. The value is capped at 50% of the resolved process/cgroup limit, and a local operator-memory value of 0 does not bypass it.
• Storage resources are mounted before startup; catalog/YAML drift has been reviewed.
• The shutdown manager sends SIGTERM and waits for a clean exit before force termination.
• Every listener reachable by a browser or otherwise untrusted client starts with --require-auth; omission is a launch-blocking error because default mode is an implicit full-SQL operator.
• The first-boot admin password comes from --admin-password-file or NYXDB_ADMIN_PASSWORD_FILE, and the mounted secret plus persistent user catalog have a recovery owner. If the source remains in the startup command it stays mounted/readable on every restart; otherwise the source is deliberately removed from the reviewed command after seeding.
• Daily operator accounts are named and non-admin; emergency admin access is narrow, recoverable, and monitored.
• Application TO ROLE grants and database-operator TO OPERATOR ROLE grants have separate owners and reviews.

Launch gate

• Static --capabilities and --readiness outputs match the approved release.
• A live protocol ping succeeds.
• A trivial query and a representative read/write smoke test succeed.
• Schema-2 negotiation, strict typed-metadata decoding, schema-1/plain downgrade on fresh sockets, and malformed-reply rejection pass against the exact client/server pair.
• A supported typed result and an unsupported/no-row fallback both prove one server execution; TSV fallback is not allowed to replay SQL.
• An unauthenticated connection can negotiate once and ping but receives authentication required for SQL; valid operator and end-user auth still succeed.
• A named non-admin operator is denied ungranted reads and all write, block-ingest, DDL, user, and grant-management attempts.
• The operator-policy boundary is explicitly tested: raw query/stream and operator-class OP_ENDPOINT_EXEC/OP_ENDPOINT_SUB enforce read or subscribe on every scanned base relation, including missing, granted, and revoked cases. Operator-class calls remain exempt from application quotas.
• An authenticated application user can execute only an approved endpoint or granted logical view and is denied raw SQL, writes, block ingest, extra claim parameters, and a same-named operator-domain grant.
• Application endpoint tests cover typed parameters, server-injected claims, lookup or live-gate revocation, static and dynamic limits, reconnect, and replacement-snapshot handling.
• Request ID and numeric query ID remain distinct in client APIs. A queued and executing one-shot can be cancelled by request ID from a separately authenticated socket, and its original terminal response is reconciled.
• Raw TCP FIN/reset and WebSocket Close cancel a representative flushed, multi-shard scan; WebSocket Ping/Pong and one pipelined request remain correct during execution.
• Exact count_distinct, uniq, and uniq_exact memory failures are fail-closed, retain scan/peak evidence, release the process-wide counter, and leave the same server process able to execute a follow-up query.
• Endpoint/view names cannot collide, or clients use relation UUIDs for unambiguous logical-view invocation; endpoint-first resolution is accepted.
• Every endpoint predicate is secure for compatible claims from any auth provider; USING provider is not treated as runtime provider affinity.
• Restart recovery has been exercised on a copy of production-like data.
• system.queries, system.query_log, system.events, system.metrics, system.parts, system.subscriptions, and system.memory_profile can be collected.
• Alerts cover process availability, connection saturation, admission queue/rejects, memory tier, WAL/flush/compaction health, stream overflow, backup failure, and storage capacity.
• A native backup completes, appears in list, and restores into an empty target.

Steady state

• Backups run on schedule and the repository is monitored independently of the source disk.
• Retention is enforced with nyxdb-backup prune only after successful restore evidence exists.
• Latest and point-in-time restore drills run on the same storage/machine class used for recovery.
• Observed RPO/RTO comes from drill artifacts, not design estimates.
• Query-log and telemetry retention are large enough for incident review but remain bounded.
• system.governance_log is streamed or exported to durable monitoring; its fixed in-engine 10,000-row drop-oldest buffer and empty-on-restart behavior are explicitly accepted.
• Application and operator grants, users, providers, endpoints, and rate limits are reviewed after every catalog or role change.
• Published-object changes have a drain procedure. Existing subscriptions are terminated before endpoint replace/drop or security/provider/claim/limit changes because they keep frozen behavior and drop removes governor state.
• Part count, committed-tail size, compaction backlog, disk usage, memory tier, query wait time, and stream lag have reviewed baselines.
• Experimental features are re-evaluated on every upgrade.

Change and recovery gate

• A verified pre-change backup exists and its manifest/frontier are recorded.
• The upgrade is scheduled as an offline restart with a user-visible outage; there is no rolling-upgrade assumption.
• Release notes explicitly permit the intended on-disk transition.
• Rollback is defined as either a compatible binary reversal or, more safely, restore into a new empty target from the pre-change backup.
• The operator will refuse to start a restore target without RESTORE_READY.
• Post-change validation covers build identity, protocol ping, catalog, representative data, streams/transforms in use, parts, memory, and backup.

See Upgrade compatibility and Backup and recovery for the executable procedures.