DB
Back homeUse case

Observability & logs

Turn raw event streams into live metrics and materialized views with continuous transforms — no external stream processor.

The problem

Why this is hard today

Observability pipelines are stream processors wearing a database costume: raw events flow into a queue, a separate processor rolls them into metrics, and yet another store serves dashboards. The rollups lag, the schemas drift, and every layer is a place for events to be double-counted or dropped.

The core operation — turn a firehose of raw events into a small set of continuously-updated metric tables — is exactly what a streaming database should do in one place.

The freshness has to come from event routing, not from a timer re-scanning the source; and the metric tables have to survive restarts without gaps.

Architecture

How NYXDB does it

Raw events append; a continuous transform rolls them into a live metric table that stays fresh through PSI routing, with gap-free recovery on restart.

  1. 01

    Ingest events

    Raw log/event lines append to a source table.

  2. 02

    Roll up (transform)

    CREATE TRANSFORM … INTO a metric table maintains counts and rates incrementally.

  3. 03

    Stay fresh (PSI)

    Only matching changes are routed to the transform — no periodic full re-scan.

  4. 04

    Serve

    Dashboards read the metric table; STREAM SELECT follows it live.

Real SQL

In practice

Metric target + rollup transform
CREATE TABLE service_errors (
  service String NOT NULL,
  errors  UInt64,
  PRIMARY KEY (service)
) DELTA (keep = 'latest');
CREATE TRANSFORM roll_errors
  INTO service_errors AS
SELECT service, count() AS errors
FROM logs
GROUP BY service;
Follow the metric live
STREAM SELECT service, errors FROM service_errors;
Inspect running transforms
SELECT name, state, lag, rows_emitted, last_error
FROM system.transforms;

Every statement follows the engine’s own test SQL shapes. See the SQL reference for full syntax.

Capabilities

What you get

In-database rollups

CREATE TRANSFORM turns raw events into live metric tables.

PSI freshness

Metrics update via event routing, not a re-scan timer.

Gap-free recovery

Transforms resume from their reflected position after restart.

Exact counts

No double-counting on the critical path.

Self-observable

system.queries, system.traces, and system.transforms expose the engine itself.

One engine

Queue, processor, and store collapse into one runtime.

Proof

Measured where it counts

gap-free

transform recovery from reflected position

ADR-080

PSI

routed freshness, not polled

core primitive

liveTODO-verify

dashboard refresh latency class

placeholder — not yet substantiated

▲Figures marked TODO-verify are placeholders pending a published, reproducible benchmark; substantiated numbers cite their source.

Learn more

Related documentation

Transforms

Standing rollup pipelines.

System tables

Observe the engine itself.

Profiling

Tracy, Parca, Pyroscope.

Roll events into live metrics

Define a transform and watch the metric table update itself.

Get started Read the docs