Profiling

Start with system.queries, system.query_log, system.traces, system.metrics, and system.events. External profiling is the next layer when those surfaces cannot attribute CPU, allocation, or a parked thread.

In-engine triage

SELECT query_id, statement, elapsed_us, queue_us, cpu_us,
       peak_memory_bytes, disk_read_us, pipeline_us,
       op_scan_us, op_filter_us, op_aggregate_us,
       op_sort_us, op_join_us
FROM system.queries
ORDER BY elapsed_us DESC;

SELECT *
FROM system.traces
ORDER BY ended_ms DESC, trace_start_epoch_us DESC, start_us DESC
LIMIT 200;

start_us is relative to the beginning of one trace; it is not globally comparable across traces. The epoch columns establish recency before ordering spans within a trace.

Capture workload rate and data scale with the profile. A flamegraph without the corresponding load and build revision is weak evidence.

Pressure-triggered heap profiles

On a compatible, armed jemalloc build, upward governor crossings request a heap profile and publish its top allocation sites:

SELECT sequence, rank, bytes, allocations, trigger, backtrace
FROM system.memory_profile
ORDER BY sequence DESC, rank ASC;

The release image starts sparse jemalloc sampling with the process:

ENV MALLOC_CONF=prof:true,prof_active:true,lg_prof_sample:19

The governor triggers dumps; it does not arm sampling. The boot probe reports the service available only when prof.active is queryable and true. Samples are not retroactive, so overriding the image with prof_active:false disables automatic first-crossing attribution even if a later prof.dump call succeeds. Override MALLOC_CONF only when a controlled investigation needs a different sample rate, and keep prof:true,prof_active:true in that replacement value.

With --data-dir, profiles are written as <data-dir>/nyxdb.<sequence>.heap. Without a data directory, the path is ./nyxdb.<sequence>.heap beneath the process working directory. Keep the exact binary, preferably an unstripped matching artifact, copy both artifacts to a controlled analysis directory, and inspect the actual absolute path with jeprof:

jeprof --text /evidence/nyxdb /evidence/nyxdb.7.heap
jeprof --lines /evidence/nyxdb /evidence/nyxdb.7.heap

Availability depends on allocator/platform build support and active-at-start sampling. An empty table or a dump warning is the designed unavailable signal. Heap files are not automatically pruned; archive incident evidence and implement bounded retention.

Tracy

Tracy zones compile out unless built with NYXDB_TRACY:

cmake -S . -B cmake-build-tracy -G Ninja \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DNYXDB_TRACY=ON \
  -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake"

cmake --build cmake-build-tracy --target nyxdb -j

The build uses on-demand collection: no capture occurs until a viewer connects. The default Tracy port is TCP 8086. Keep it reachable only through the controlled diagnostic path. For headless capture:

tracy-capture -a <server-host> -o /tmp/nyxdb.tracy

NYXDB_TRACY_MEMORY is a separate opt-in and adds allocation-hook overhead while attached. Use it only for a focused investigation.

Parca

The engine repository provides an observability/ compose stack for a local reproduction:

cd observability
docker compose config
NYXDB_IMAGE=<exact-image> docker compose up -d

The agent needs a native Linux host, BPF/perf support, host process visibility, and narrowly scoped elevated privileges. Pin the agent/server images and scope collection to the NYXDB cgroup.

For a wedged thread, collect a debugger thread dump or use node-level bcc offcputime / perf lock contention according to the host runbook. Attach only long enough to capture evidence, and expect debugger attachment to pause or perturb the process.

The complete, revision-matched procedure is the engine repository's profiling runbook.